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General Guidelines Overview 


Welcome to the Search Quality Rating Program! 

As a Search Quality evaluator, you will work on many different types of rating projects. The General Guidelines primarily 
cover Page Quality (PQ) rating and Needs Met (NM) rating; however, the concepts are also important for many other 
types of rating tasks. 


For brevity, we refer to “Search Quality Evaluators” as “raters” in these guidelines. 


Should | buy an iPhone? 


Google 


Google Search I'm Feeling Lucky 


Is this page trustworthy? 


Buy cheap iPhones for cheap - BuyCheapiphones.123 
www.buycheapphones.123/buy-cheap-cheaper 
Browse and buy iPhones on the web's number #1 cheap iPhone marketplace. 


Register for FREE! Get cheap iPhones here. 
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0.0 Introduction to Search Quality Rating 
0.1 The Purpose of Search Quality Rating 


Your ratings will be used to evaluate search engine quality around the world. Good search engines give results that are 
helpful for users in their specific language and locale. 


It is important that you are familiar with and comfortable using a search engine. We encourage you to be an expert in 
Google search! For example, experiment with using operators (e.g., quotes or a dash) in your searches or try using 
Google’s advanced search option. 


0.2 Raters Must Represent the User 


It is very important for you to represent users in the locale you evaluate. You must be very familiar with the task language 
and location in order to represent the experience of users in your locale. If you do not have the knowledge to do this, 
please inform your vendor. 


0.3 Browser Requirements 


Check with your vendor for browser requirements. You may use helpful browser add-ons or extensions, but please do not 
use add-ons or extensions that interfere with or alter the user experience of the page. 


0.4 Ad Blocking Extensions 


Do not use add-ons or extensions that block ads for Needs Met rating or Page Quality rating. These add-ons or 
extensions may cause you to give incorrect ratings. As a rater, only use an ad blocking extension or add-on if specifically 
instructed to do so in the project-specific instructions. 


0.5 Internet Safety Information 


In the course of your work, you will visit many different webpages. Some of them may harm your computer unless you are 
careful. Please do not download any executables, applications, or other potentially dangerous files, or click on any links 
that you are uncomfortable with. 


It is strongly recommended that you have antivirus and antispyware protection on your computer. This software 
must be updated frequently or your computer will not be protected. There are many free and for-purchase 
antivirus and antispyware products available on the web. 


See here for a Wikipedia page on antivirus software and here for a Wikipedia page on spyware. 


We suggest that you only open files with which you are comfortable. The file formats listed below are generally 
considered safe if antivirus software is in place. 


txt (text file) 

.ppt or .pptx (Microsoft PowerPoint) 
.doc or .docx (Microsoft Word) 

.xls or .xlsx (Microsoft Excel) 

.pdf (PDF) files 


If you encounter a page with a warning message, such as “Warning-visiting this web site may harm your computer,” or if 
your antivirus software warns you about a page, you should not try to visit the page to assign a rating. 


You may also encounter pages that require RealPlayer or the Adobe Flash plug-in. These are generally safe to download. 
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Part 1: Page Quality Rating Guideline 


1.0 Introduction to Page Quality Rating 


A Page Quality (PQ) rating task consists of a URL and a grid to record your observations, in order to guide your 
exploration of the landing page and the website associated with the URL. Ultimately, the goal of Page Quality rating is to 
evaluate how well the page achieves its purpose. Because different types of websites and webpages can have very 
different purposes, our expectations and standards for different types of pages are also different. 


Here's what you'll need to be a successful Page Quality rater: 


e Your experience using the web as an ordinary user in your rating locale. 
e In-depth knowledge of these guidelines. 
e And most importantly—practice doing PQ rating tasks! 


The examples in these guidelines are very important. Please view each one and keep in mind a few notes about the 
examples: 


e Webpages and websites change rapidly, so we use images or "snapshots" of webpages in most of our examples. 

e The information in the examples was accurate at the time it was added, but content and websites may change 
over time. 

e Some examples show pages on desktop and some show pages on mobile devices. 


2.0 Understanding Webpages and Websites 


PQ rating requires an in-depth understanding of websites. We'll start with the basics. Along the way, we'll share 
important information about Page Quality rating, so please read through this section even if you are a website expert! 


2.1 Important Definitions 
Here are some important definitions: 


A webpage is connected to the World Wide Web and can be viewed or "visited" using a web browser (e.g., Chrome), a 
browser on your phone, or a search app. In the 1990s, webpage content was mostly text and links. Today, webpage 
content includes many forms of media (such as images, videos, etc.) and functionality (such as online shopping features, 
email, calculator functionality, online games, etc.). 


A URL is a character string that your web browser uses to “find” and display a webpage. Page Quality rating doesn't 
require you to have in-depth understanding of the structure of URLs, i.e., you don’t need to know the difference between 
host, domain, etc. But if you are interested, see here to read more. 


A website or site is a group of World Wide Web pages usually containing hyperlinks to each other and made available 
online by an individual, company, educational institution, government, or organization. Popular websites include 
Facebook, Wikipedia, Yahoo, YouTube, etc. 


Note: In these guidelines, we will use the word “website” to refer to a collection of pages owned and controlled by a single 
entity (individual, business, etc.). But we will also use “website” to refer to major “independent” sections (or hosts) of 
some websites that were created to achieve separate purposes. For example, the Yahoo website is organized into 
different sections (or hosts), such as Yahoo Finance (finance.yahoo.com), Yahoo Mail (mail.yahoo.com), Yahoo Sports 
(sports.yahoo.com), etc. Each of these has its own purpose. It’s OK to refer to each of these sections as a website; for 
example, the Yahoo Finance website and the Yahoo Sports website. You may also refer to pages on Yahoo Finance or 
Yahoo Sports as belonging to the Yahoo website. 
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A homepage of a website is the main page of the site. It is usually the first page that users see when the site loads. For 
example, http:/Awww.apple.com is the homepage of the Apple site, http://www.yahoo.com is the homepage of the Yahoo 
company site, and http://finance.yahoo.com is the homepage of Yahoo Finance. You can usually find the homepage of a 
website by clicking on a “home” link or logo link on subpages of a website. 


A subpage on a website is any page on the site other than the homepage. For example, http://www.apple.com/iphone is 
a subpage on the Apple website, and http://finance.yahoo.com/options is a subpage on the Yahoo Finance website. 


A webmaster is the person who is responsible for maintaining a website. 


Important: You must be very comfortable exploring websites, both by clicking links and modifying URLs in the address 
bar of your web browser. Become a website detective and explorer! 


2.2 What is the Purpose of a Webpage? 


The purpose of a page is the reason or reasons why the page was created. Every page on the Internet is created for a 
purpose, or for multiple purposes. Most pages are created to be helpful for users. Some pages are created merely to 
make money, with little or no effort to help users. Some pages are even created to cause harm to users. The first step in 
understanding a page is figuring out its purpose. 


Why is it important to determine the purpose of the page for PQ rating? 


e The goal of PQ rating is to determine how well a page achieves its purpose. In order to assign a rating, you must 
understand the purpose of the page and sometimes the website. 

e By understanding the purpose of the page, you'll better understand what criteria are important to consider when 
evaluating that particular page. 

e Websites and pages should be created to help users. Websites and pages that are created with intent to harm 
users, deceive users, or make money with no attempt to help users, should receive the Lowest PQ rating. More 
on this later. 


As long as the page is created to help users, we will not consider any particular page purpose or type to be higher quality 
than another. For example, encyclopedia pages are not necessarily higher quality than humor pages. 


Important: There are highest quality and lowest quality webpages of all different types and purposes: shopping pages, 
news pages, forum pages, video pages, pages with error messages, PDFs, images, gossip pages, humor pages, 
homepages, and all other types of pages. The type of page does not determine the PQ rating—you have to understand 
the purpose of the page to determine the rating. 


Common helpful page purposes include (but are not limited to): 
To share information about a topic. 

To share personal or social information. 

To share pictures, videos, or other forms of media. 

To express an opinion or point of view. 

To entertain. 

To sell products or services. 

To allow users to post questions for other users to answer. 
To allow users to share files or to download software. 


Here are a few examples where it is easy to understand the purpose of the page: 


Type of Page Purpose of the Page 
News website homepage To inform users about recent or important events. 
O sell or give information about the product. 


Video page To allow users to watch a video. 
Currency converter page To calculate equivalent amounts in different currencies. 
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Here are two examples of helpful pages where the purpose of the page is not as obvious: 


Page with a 
Non-Obvious Purpose 


This page looks as though the purpose is to share factual information, but the page starts 
with the text “Christopher Columbus was born in 1951 in Sydney, Australia.” This is 
obviously inaccurate! Was this page created to help users or to trick and confuse users? 


In this case, exploring the website can help us understand the purpose of the page. This 


i . . . . . . 
Christopher Columbus website was built by educators to teach about interpreting information found on the Internet. 


Page Example 


After reading about the website on the About This Site page, it should be clear that the 
purpose of the page is to serve as an educational tool. The information on the page is 
deliberately inaccurate so that it can be used as an example of misinformation on the 
Internet. This page and website do have a helpful and beneficial purpose. 


At first glance, this page may seem pointless or strange. However, it is a page from a 
humorous site that encourages users to post photos with mouths drawn on them. The 
purpose of the page is humor or artistic expression. This page has a helpful or beneficial 
purpose. 


OmNomNomNom Page 
Example 


Even though the About page on this website is not very helpful, the website explains itself 
on its FAQ page. 


2.3 Your Money or Your Life (YMYL) Pages 


Some types of pages could potentially impact the future happiness, health, or financial stability of users. We call such 
pages “Your Money or Your Life” pages, or YMYL. The following are examples of YMYL pages: 


e Shopping or financial transaction pages: webpages that allow users to make purchases, transfer money, pay 
bills, etc. online (Such as online stores and online banking pages). 

e Financial information pages: webpages that provide advice or information about investments, taxes, retirement 
planning, home purchase, paying for college, buying insurance, etc. 

e Medical information pages: webpages that provide advice or information about health, drugs, specific diseases 
or conditions, mental health, nutrition, etc. 

e Legal information pages: webpages that provide legal advice or information on topics such as divorce, child 
custody, creating a will, becoming a citizen, etc. 

e News articles or public/official information pages important for having an informed citizenry: webpages 
that include information about local/state/national government processes, people, and laws; disaster response 
services; government programs and social services; news about important topics such as international events, 
business, politics, science, and technology; etc. Please use your judgment and knowledge of your locale. Keep 
in mind that not all news articles are necessarily considered YMYL. 

e Other: there are many other topics that you may consider YMYL, such as child adoption, car safety information, 
etc. Please use your judgment. 


We have very high Page Quality rating standards for YMYL pages because low quality YMYL pages could potentially 
negatively impact users’ happiness, health, or financial stability. 


2.4 Understanding Webpage Content 


All of the content on a webpage can be classified as one of the following: Main Content (MC), Supplementary Content 
(SC), or Advertisements/Monetization (Ads). In order to understand the purpose of a webpage and do PQ rating, you will 
need to be able to distinguish among these different parts of the page. 


Webpage design can be complicated, so make sure to click around and explore the page. See what kind of content is 
behind the tabs and test out the interactive page features. Content behind the tabs may be considered part of the MC, 
SC, or Ads, depending on what the content is. 
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2.4.1 Identifying the Main Content (MC) 


Main Content is any part of the page that directly helps the page achieve its purpose. Webmasters directly control the MC 
of the page (except for user-generated content). MC can be text, images, videos, page features (e.g., calculators, 
games), or it can be user-generated content such as videos, reviews, articles, etc. that users have added or uploaded to 
the page. Note that tabs on some pages lead to even more information (e.g., customer reviews) and can sometimes be 
considered part of the MC of the page. 


Type of Page and Purpose MC Highlighted in Yellow 
News website homepage: the purpose is to inform users about recent or important events. MC - News Homepage 
News article page: the purpose is to communicate information about an event or news topic. MC - News Article 


Store product page: the purpose is to sell or give information about the product. 
e Content behind the Reviews, Shipping, and Safety Information tabs are considered to be |MC - Shopping Page 
part of the MC. 


urrency converter page: the purpose is to calculate equivalent amounts in different currencies MC - Currency Converter 


Video page: the purpose is to allow users to view a video. MC - Video Page 
Blog post page: the purpose is to display a blog post. MC - Blog Post Page 


earch engine homepage: the purpose is to allow users to enter a query and search the Internet. |MC - Search Engine Homepage 
Bank login page: the purpose is to allow users to log in to bank online MC - Bank Login Page 


2.4.2 Identifying the Supplementary Content (SC) 


Supplementary Content contributes to a good user experience on the page, but does not directly help the page achieve its 
purpose. SC is controlled by webmasters and is an important part of the user experience. One common type of SC is 
navigation links that allow users to visit other parts of the website. Note that in some cases, content behind tabs may be 
considered part of the SC of the page. 


Sometimes the easiest way to identify SC is to look for the parts of the page that are not MC or Ads. 


Type of Page and Purpose SC Highlighted in Blue 
News article page: the purpose is to communicate information about an event or news topic. SC - News Article 
Store product page: the purpose is to sell or give information about the product. SC - Shopping Page 


Video page: the purpose is to allow users to view a video. SC - Video Page 
Blog post page: the purpose is to display a blog post. SC - Blog Post Page 


2.4.3 Identifying Advertisements/Monetization (Ads) 


Ads may contribute to a good user experience. Advertisements/Monetization (Ads) is content and/or links that are 
displayed for the purpose of monetizing (making money from) the page. The presence or absence of Ads is not by itself a 
reason for a High or Low quality rating. Without advertising and monetization, some webpages could not exist because it 
costs money to maintain a website and create high quality content. 


There are several different ways to monetize a webpage, including advertisements and affiliate programs. See here for 
more information on website monetization. Note that monetization on mobile pages may be more subtle than 
monetization on desktop pages. 

The most common type of monetization is advertisements. Ads may be labeled as "ads," "sponsored links," “sponsored 
listings,” “sponsored results,” etc. Usually, you can click on the links or mouse over the content to determine whether they 
are Ads, as they often refer to a URL outside of that website. Ads may change when you reload the page, and different 
users may see different Ads on the same page. 


Webmasters can choose to display Ads on their page (such as by joining an advertising network), but they may not 


always directly control the content of the Ads. However, we will consider a website responsible for the overall quality of 
the Ads displayed. 
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Important: For the purpose of this guideline, we will consider monetized links of any type to be “Ads.” See here for 
different types of website monetization. 


Type of Page and Purpose Ads Highlighted in Red 
News article page: the purpose is to communicate information about an event or news topic. Ads - News Article 


Video page: the purpose is to allow users to view a video. Ads - Video Page 
Blog post page: the purpose is to display a blog post. Ads - Blog Post Page 
Store product page: the purpose is to sell or give information about the product. No ads — Shopping Page 


2.4.4 Summary of the Parts of the Page 
Let's put it all together. 


e Main Content (MC) is any part of the page that directly helps the page achieve its purpose. MC is (or should be!) the 
reason the page exists. The quality of the MC plays a very large role in the Page Quality rating of a webpage. 

e Supplementary Content (SC) is also important. SC can help a page better achieve its purpose or it can detract from 
the overall experience. 

e Many pages have advertisements/monetization (Ads). Without advertising and monetization, some webpages 
could not exist because it costs money to maintain a website and create high quality content. The presence or 
absence of Ads is not by itself a reason for a High or Low quality rating. 


On some pages, reviews may be considered MC, and on other pages they may be considered SC. Use your best 
judgment and think about the purpose of the page. 


Do not worry too much about identifying every little part of the page. Think about which parts of the page are the MC. 
Next, look for the Ads. Anything left over can be considered SC. 


Type of Page and Purpose MC, SC, and Ads Highlighted 
News article page: the purpose is to communicate information about an event or news topic Summary - News Article 
Store product page: the purpose is to sell or give information about the product Summary - Shopping Page 


Currency converter page: the purpose is to calculate equivalent amounts in different currencies ummary - Currency Converter 
Blog post page: the purpose is to display a blog post Summary - Blog Post Page 
Bank login page: the purpose is to allow users to log in to bank online Summary - Bank Login Page 


Video page: the purpose is to allow users to view a video. Summary - Video Page 
; : a ies. |S 


2.5 Understanding the Website 


Pages often make more sense when viewed as part of a website. Some of the criteria in Page Quality rating are based 
on the website the page belongs to. 


In order to understand a website, look for information about the website on the website itself. Websites are usually very 
eager to tell you all about themselves! 


You must also look for reputation information about the website. We need to find out what outside, independent sources 
say about the website. When there is disagreement between what the website says about itself and what reputable 
independent sources say about the website, we'll trust the independent sources. 


2.5.1 Finding the Homepage 


The homepage of a website usually contains or has links to important information about the website. Webmasters usually 
make it easy to get to the homepage of the website from any page on the site. 
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Here's how to find the homepage of a website: 


e Examine the landing page of the URL in your PQ rating task. 

Find and click on the link labeled “home” or “main page.” 

e Having trouble finding it? Try using “Ctrl-F” (“command-F” on a Mac) to search the page for the text “home” or 
“main.” You may also try clicking on the website logo, which is usually at the top of the page. 


Sometimes, you may be given a webpage or website that appears to have no navigation links, no homepage link, and no 
logo or other means to find the homepage. Even some High or Highest quality pages lack a way to navigate to the 
homepage. If you can't find a link to the homepage, modify the URL by removing everything to the right of “.com,” “.org,” 
“net,” “info,” etc. and refresh the page. 


Occasionally, your rating task will include a URL for which there are two or more justifiable “homepage” candidates. For 
example, you may not be sure whether the homepage of the URL http://finance.yahoo.com/news/category-stocks is 
http://finance.yahoo.com or_http://www.yahoo.com. 


Important: When you have more than one homepage “candidate,” please use whichever one offers the most information 
about the specific webpage in the rating task. Use your judgment. The goal is to understand the webpage and the 
website(s) it is associated with, not find the one unique, correct homepage. 


In the following examples, we have included the URL of the page to be evaluated in the rating task, as well as the URL of 
its associated homepage. We have also included an image that shows where to click on the landing page to navigate to 
the homepage. In the image, you will see a red box around the link or logo you would click to navigate to the homepage. 


URL of the Task Page 


http://www.williams-sonoma.c 
om/products/shun-premier-7- 
piece-knife-block-set 


http://answers.yahoo.com/qu 
estion/index;_ylt=AnAYEU1f 
ED6ncg1jRCFy30kk5XNG;_y 
lv=3?qid=20091214193523A 
AQqHQS 


http://nhms.harvard.edu/about- 
hms/facts-figures 


http://www. williams-sonoma.com 


http://answers.yahoo.com 


In this case, we will consider 
http://answers.yahoo.com the homepage, rather 
than http://www.yahoo.com. Why? Because 
clicking on the logo takes the user to 
http://answers.yahoo.com. In addition, 
http://answers.yahoo.com has information about 
the Yahoo Answers website. It is very difficult to 
find specific information about 
http://answers.yahoo.com on the 
http:/www.yahoo.com homepage. 


http://hms.harvard.edu 


In this case, we will consider the Harvard Medical 
School page at http://hms.harvard.edu to be the 
homepage, rather than http://www.harvard.edu 
(which is the homepage of Harvard University). 
Clicking the logo at the top of 
http://nms.harvard.edu/about-hms/facts-figures 
takes users to http://hms.harvard.edu, not to 
http:/Awww.harvard.edu. 
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7 Image that shows where to click to get to the 


Williams-Sonoma Homepage 


WILLIAMS 
SONOMA 


This “WILLIAMS-SONOMA” logo shown in the 
upper center of the page is clickable and takes 
users to the homepage of the website. 


Specific Yahoo Answers Page 


YAHOO! Answers 


This “YAHOO ANSWERS?’ logo in the upper left 
part of the page is clickable and takes users to 
the homepage of the website. 


Harvard Medical School Facts and Figures Page 


ey HARVARD 


MEDICAL SCHOOL 


This “Harvard Medical School” logo in the upper 
left part of the page is clickable and takes users 
to the homepage of the Harvard Medical School 
website. 
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2.5.2 Finding Who is Responsible for the Website and Who Created the Content on the Page 
Every page belongs to a website, and it should be clear: 


e Who (what individual, company, business, foundation, etc.) is responsible for the website. 
e Who (what individual, company, business, foundation, etc.) created the content on the page you are evaluating. 


Websites are usually very clear about who created the content on the page. There are many reasons for this: 


Commercial websites may have copyrighted material they want to protect. 

Businesses want users to know who they are. 

Artists, authors, musicians, and other original content creators usually want to be known and appreciated. 
Foundations often want support and even volunteers. 

High quality stores want users to feel comfortable buying online. 


Most websites have “contact us” or “about us” or “about” pages that provide information about who owns the site. Many 
companies have an entire website or blog devoted to who they are and what they are doing, what jobs are available, etc. 
Google and Marriott are both examples of this, and there are many others: 


Google Official Blog 
Marriott Blog 
Southwest Airlines Blog 
Netflix Tech Blog 


Often a business or organization is responsible for the content of a website, not an individual person. The IBM 
Corporation is responsible for the content on ibm.com. The Cleveland Clinic is responsible for the content on 
clevelandclinic.org. An individual is not responsible for the content on these websites, even though many individuals 
contributed to creating and maintaining the content. In these cases, we will view the business or organization as 
responsible for the content on every single page, as well as maintenance of the website. 


On some websites, users create the MC of many pages, while the business or organization itself maintains the website. 
The company Facebook is responsible for the Facebook website, but individuals create the content on their personal 
Facebook pages. The company Wikipedia is responsible for the Wikipedia website, but individuals create article content. 
Other websites with user-generated content include YouTube, Twitter, other social networking websites, other article 
publishing websites, Q&A websites, forums, etc. For these websites, you must look at each page to determine the 
author(s) or creator(s) of the content on that page. 


Finally, there are some websites that show licensed or syndicated content. This means that the website has paid money 
or has some business relationship with the creator of the content. In these cases, we will consider the website itself to be 
responsible for the licensed or syndicated content, even if it wasn’t created by the website. 


2.5.3 Finding About Us, Contact Information, and Customer Service Information 


Many websites are interested in communicating with their users. There are many reasons that users might have for 
contacting a website, from reporting problems such as broken pages, to asking for content removal. Many websites offer 
multiple ways for users to contact the website: email addresses, phone numbers, physical addresses, web contact forms, 
etc. Sometimes, this contact information is even organized by department and provides the names of individuals to 
contact. 
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The types and amount of contact information needed depend on the type of website. Contact information and customer 
service information are extremely important for websites that handle money, such as stores, banks, credit card 
companies, etc. Users need a way to ask questions or get help when a problem occurs. 


For shopping websites, we'll ask you to do some special checks. Look for contact information—including the store’s 
policies on payment, exchanges, and returns. Sometimes this information is listed under “customer service.” 


Some kinds of websites need fewer details and a smaller amount of contact information for their purpose. For example, 
humor websites may not need the level of detailed contact information we would expect from online banking websites. 


Occasionally, you may encounter a website with a legitimate reason for anonymity. For example, personal websites may 
not include personal contact information such as an individual’s home address or phone number. Similarly, websites with 
user-generated content may allow the author to identify him/herself with an alias or username only. 


To find contact or customer service information for a website, start with the homepage. Look for a “contact us” or 
“customer service” link. Explore the website if you cannot find a “contact us” page. Sometimes you will find the contact 
information on a “corporate site” link or even on the company’s Facebook page. Be a detective! 


Note that different locales may have their own specific standards and requirements for what information should be 
available on the website. 


2.6 Website Reputation 


A website's reputation is based on the experience of real users, as well as the opinion of people who are experts in the 
topic of the website. Keep in mind that websites often represent real companies, organizations, and other entities. 
Therefore, reputation research applies to both the website and the actual company, organization, or entity that the website 
is representing. 


A website's reputation can also help you understand what a website is best known for, and as a result how well it 
accomplishes its purpose. For example, newspapers may be known for high quality, independent investigative reporting 
while satire websites may be known for their humor. 


Many websites are eager to tell users how great they are. Some webmasters have read these rating guidelines and write 
“reviews” on various review websites. But for Page Quality rating, you must also look for outside, independent reputation 

information about the website. When the website says one thing about itself, but reputable external sources disagree with 
what the website says, trust the external sources. 


Your job is to truly evaluate the Page Quality of the site, not just blindly accept information on one or two pages of the 
website. Be skeptical of claims that websites make about themselves. 


Copyright 2017 14 


2.6.1 Reputation Research 


Use reputation research to find out what real users, as well as experts, think about a website. Look for reviews, 
references, recommendations by experts, news articles, and other credible information created/written by individuals 
about the website. 


Stores frequently have user ratings, which can help you understand a store’s reputation based on the reports of people 
who actually shop there. We consider a large number of positive user reviews as evidence of positive reputation. 


Many other kinds of websites have reputations as well. For example, you might find that a newspaper website has won 
journalistic awards. Prestigious awards, such as the Pulitzer Prize award, are strong evidence of positive reputation. 


When a high level of authoritativeness or expertise is needed, the reputation of a website should be judged on what 
expert opinions have to say. Recommendations from expert sources, such as professional societies, are strong evidence 
of very positive reputation. 


Reputation research is necessary for all websites you encounter. Do not just assume websites you personally use have a 
good reputation. Please do research! You might be surprised at what you find. 


2.6.2 Sources of Reputation Information 


Look for information written by a person, not statistics or other machine-compiled information. News articles, Wikipedia 
articles, blog posts, magazine articles, forum discussions, and ratings from independent organizations can all be sources 
of reputation information. Look for independent, credible sources of information. 


Sometimes, you will find information about a website that is not related to its reputation. For example, pages like Alexa 
have information about Internet traffic to the website, but do not provide evidence of positive or negative reputation. You 
can ignore this information since it's not helpful for Page Quality rating. 


2.6.3 Customer Reviews of Stores/Businesses 


Customer reviews can be helpful for assessing the reputation of a store or business. However, you should interpret these 
reviews with care, particularly if there are only a few. Be skeptical of both positive and negative user reviews. Anyone can 
write them, including the creator of the website or someone the store or business hires for this purpose. See here fora 
New York Times article on fake reviews and here for a Guardian article on fake reviews. 


When interpreting customer reviews, try to find as many as possible. Any store or website can get a few negative 
reviews. This is completely normal and expected. Large stores and companies have thousands of reviews and most 
receive some negative ones. 


It is also important to read the reviews because the content of the reviews matter, not just the number. Credible, 
convincing reports of fraud and financial wrongdoing is evidence of extremely negative reputation. A single encounter 
with a rude clerk or the delayed receipt of a single package should not be considered negative reputation information. 
Please use your judgment. 
2.6.4 How to Search for Reputation Information 
Here is how to research the reputation of the website: 

1. Identify the “homepage” of the website. For example, for the IBM website, ibm.com is the homepage. 

2. Using ibm.com as an example, try one or more of the following searches on Google: 
[ibm -site:ibm.com]: A search for IBM that excludes pages on ibm.com. 
[“ibm.com” -site:ibm.com]: A search for “ibm.com” that excludes pages on ibm.com. 


[ibm reviews -site:ibm.com] A search for reviews of IBM that excludes pages on ibm.com. 
[“ibm.com” reviews -site:ibm.com]: A search for reviews of “ibm.com” that excludes pages on ibm.com. 
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Note: When searching for reputation information, try to find sources that were not written or created by the 
website or company itself. For example, IBM might have official Facebook or Twitter pages that it closely 
maintains, which would not be considered independent sources of reputation information about the company. 
See here for a Wikipedia article on identifying and using independent sources. 


3. Look for articles, reviews, forum posts, discussions, etc. written by people about the website. For businesses, 
there are many sources of reputation information and reviews. Here are some examples: Yelp, Better Business 
Bureau (a nonprofit organization that focuses on the trustworthiness of businesses and charities), Amazon, and 
Google Shopping. You can try searching on specific sites to find reviews. For example, you can try [ibm 
site:bbb.org] or [“ibm.com” site:bbb.org]. 


Note: You will sometimes find high ratings on the Better Business Bureau (BBB) website because there is very 
little data on the business, not because the business has a positive reputation. However, very low ratings on BBB 
are usually the result of multiple unresolved complaints. Please consider very low ratings on the BBB site to be 
evidence for a negative reputation. 


4. See if there is a Wikipedia article or news article from a well-known news site. Wikipedia can be a good source of 
information about companies and organizations. For example, try [ibm site:en.wikipedia.org] or [“ibm.com” 
site:en.wikipedia.org]. News articles and Wikipedia articles can help you learn about a company and may include 
information specific to reputation, such as awards and other forms of recognition, or also controversies and 
issues. Note that some Wikipedia articles include a message warning users that there are disagreements on 
some of the content, or that the content may be outdated. This may be an indication that additional research is 
necessary. 


Here are some examples of reputation information: 


Positive reputation information: Users in the U.S. 
can obtain free credit reports on this website by 
providing their Social Security Number. Note that 
the Wikipedia article tells us that 
“AnnualCreditReport.com is the only federally 
mandated and authorized source for obtaining a 
free credit report.” 


Search results for [annualcreditreport.com 
-site:annualcreditreport.com 


annualcreditreport.com Wikipedia article about annualcreditreport.com 


Wall Street Journal article about 


annualcredifrenort Com Note: Almost every website will have complaints 


about customer service, so it is important to look at 
various sources and reviews in your reputation 
research. 


Positive reputation information: According to 
Wikipedia, the Cleveland Clinic “is currently 

PES F a regarded as one of the top 4 hospitals in the United 
Wikipedia article about clevelandclinic.org States as rated by U.S. News & World Report,” 
which you will also find in the article on the best 
hospitals in the U.S. Users can trust medical 
information on this website. 


Search results for [clevelandclinic.org 


clevelandclinic.org 


US News & World Report article about the best 
hospitals in the U.S. 


Positive reputation information: Notice the 
Search results for [csmonitor.com highlighted section in the Wikipedia article about 
-site:csmonitor.com The Christian Science Monitor newspaper, which 
csmonitor.com tells us that the newspaper has won seven Pulitzer 
Wikipedia article about The Christian Science Prize awards. From this information, we can infer 
Monitor that the csmonitor.com website has a positive 
reputation. 
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Positive reputation information: We learn in the 

Wikipedia article that “Kernel.org is a 

main repository of source code for the Linux kernel, 
Search results for [kernel.org —site:kernel.org the base of the popular Linux operating system. It 

kernel.org Wildaedia arice about Kemalon makes all versions of the source code available to all 

Derai eh 6 abo see ord users. It also hosts various other projects, 

like Google Android. The main purpose of the site is 

to host a repository for Linux kernel developers and 

maintainers of Linux distributions.” 


Search to find reputation information Extremely negative reputation information: This 
Search to find reviews business has a BBB rating of F (i.e., lowest rating 


Sitesdlino chidren: given by BBB). There is a news article about 
H H . . . 
Ste Seng Sere BBB negative review financial fraud. There are many reviews on websites 


jungle gym ae A ae 
TrustLink negative reviews describing users sending money and not receiving 
anything from various sources. 
Negative news article 


Search to find reputation information 
, i Extremely negative/malicious reputation 
gp pag ; n N : cara 
A a n BBB page information: This website engaged in criminal 
Wikipedia article behavior such as physically threatening users. 


New York Times article 


Search to find scams related this organization 


Organization serving the [Negative review 1 Extremely negative reputation information: There 
g g Regau ve review i : f ¢ ; 
are many detailed negative articles on news sites 


i i i H . . . . . 
hospltaized veteran G veteran Negative review 2 and charity watchdog sites about this organization 
gomeuan. Negative review 3 describing fraud and financial mishandling. 


Negative review 4 


2.6.5 What to Do When You Find No Reputation Information 

You should expect to find reputation information for large businesses and websites of large organizations. 

Frequently, you will find little or no information about the reputation of a website for a small organization. This is not 
indicative of positive or negative reputation. Many small, local businesses or community organizations have a small “web 


presence” and rely on word of mouth, not online reviews. For these smaller businesses and organizations, lack of 
reputation should not be considered an indication of low page quality. 


3.0 Overall Page Quality Rating Scale 
The overall Page Quality rating scale offers five rating options: Lowest, Low, Medium, High, and Highest. 
On Page Quality rating tasks, you will use the Page Quality sliding scale (slider) to assign the overall PQ rating. The 


slider looks like this: 


Lowest Lowest+ Low Low+ Medium Medium+ High High+ Highest 


Copyright 2017 17 


You may also use the in-between ratings of Lowest+, Low+, Medium+, and Hight. Please interpret the “+” as “+ %,” 
meaning that the Lowest+ rating is halfway between Lowest and Low, Low+ is halfway between Low and Medium, etc. 


In the following sections, you will learn about characteristics of Lowest, Low, Medium, High, and Highest quality pages. 


3.1 Page Quality Rating: Most Important Factors 


Here are the most important factors to consider when selecting an overall Page Quality rating: 


e Expertise, Authoritativeness, Trustworthiness: This is an important quality characteristic. Use your research 
on the additional factors below to inform your rating. 

e Main Content Quality and Amount: The rating should be based on the landing page of the task URL. 

e Website Information/information about who is responsible for the website: Links to help with website 
information research will be provided. 

e Website Reputation: Links to help with reputation research will be provided. 


Note: Some tasks may ask you to view the page on your phone, but to do research (e.g., finding website information and 
reputation) on your desktop. Other tasks may ask you to do everything on desktop. Please follow instructions in the task. 


3.2 More about Expertise, Authoritativeness, and Trustworthiness (E-A-T) 


The amount of expertise, authoritativeness, and trustworthiness (E-A-T) that a webpage/website has is very important. 
MC quality and amount, website information, and website reputation all inform the E-A-T of a website. 


Keep in mind that there are “expert” websites of all types, even gossip websites, fashion websites, humor websites, forum 
and Q&A pages, etc. In fact, some types of information are found almost exclusively on forums and discussions, where a 
community of experts can provide valuable perspectives on specific topics. 


e High quality medical advice should come from people or organizations with appropriate medical expertise or 
accreditation. High quality medical advice or information should be written or produced in a professional style and 
should be edited, reviewed, and updated on a regular basis. 

e High quality news articles should contain factually accurate content presented in a way that helps users achieve a 
better understanding of events. Established editorial policies and review processes are typically held by high 


quality news sources (example 1, example 2). 

e High quality information pages on scientific topics should represent well-established scientific consensus on 
issues where such consensus exists. 

e High quality financial advice, legal advice, tax advice, etc., should come from expert sources and be maintained 
and updated regularly. 

e High quality advice pages on topics such as home remodeling (which can cost thousands of dollars and impact 
your living situation) or advice on parenting issues (which can impact the future happiness of a family) should also 
come from “expert” or experienced sources that users can trust. 

e High quality pages on hobbies, such as photography or learning to play a guitar, also require expertise. 


Some topics require less formal expertise. Many people write extremely detailed, helpful reviews of products or 
restaurants. Many people share tips and life experiences on forums, blogs, etc. These ordinary people may be 
considered experts in topics where they have life experience. If it seems as if the person creating the content has the 
type and amount of life experience to make him or her an “expert” on the topic, we will value this “everyday expertise” and 
not penalize the person/webpage/website for not having “formal” education or training in the field. 


It’s even possible to have everyday expertise in YMYL topics. For example, there are forums and support pages for 
people with specific diseases. Sharing personal experience is a form of everyday expertise. Consider this example. 
Here, forum participants are telling how long their loved ones lived with liver cancer. This is an example of sharing 
personal experiences (in which they are experts), not medical advice. Specific medical information and advice (rather 
than descriptions of life experiences) should come from doctors or other health professionals. 


Think about the topic of the page. What kind of expertise is required for the page to achieve its purpose well? The 
standard for expertise depends on the topic of the page. 
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4.0 High Quality Pages 


In this section, we will describe characteristics of High quality pages and give many examples. Examples help calibrate 
your ratings, so please review each one. 


4.1 Characteristics of High Quality Pages 


High quality pages are satisfying and achieve their purpose well. High quality pages exist for almost any purpose, from 
giving information to making you laugh. 


What makes a High quality page? A High quality page may have the following characteristics: 


e High level of Expertise, Authoritativeness, and Trustworthiness (E-A-T), including the E-A-T of the publisher 
and/or individual author for news articles and information pages on YMYL topics. 

e A satisfying amount of high quality MC. 

e Satisfying website information and/or information about who is responsible for the website or satisfying customer 
service information, if the page is primarily for shopping or includes financial transactions. 

e Positive website reputation for a website that is responsible for the MC on the page. 


4.2 A Satisfying Amount of High Quality Main Content 


The quality of the MC is one of the most important criteria in Page Quality rating, and informs the E-A-T of the page. For 
all types of webpages, creating high quality MC takes a significant amount of at least one of the following: time, effort, 
expertise, and talent/skill. For news articles and information pages, high quality MC must be factually accurate for the 
topic and must be supported by expert consensus where such consensus exists. 


For each page you evaluate, spend a few minutes examining the MC before drawing a conclusion about it. Read the 
article, watch the video, examine the pictures, use the calculator, play the online game, etc. Remember that MC also 
includes page features and functionality, so test the page out. For example, if the page is a product page on a store 
website, put at least one product in the cart to make sure the shopping cart is functioning. If the page is an online game, 
spend a few minutes playing it. 


The purpose of the page will help you determine what high quality content means for that page. For example, High quality 
information pages should be factually accurate, clearly written, and comprehensive. High quality shopping content should 
allow users to find the products they want and to purchase the products easily. High quality humor or satire should be 
entertaining, while factual accuracy is not a requirement as long as the page would be understood as satire by users. 


The amount of content necessary for the page to be satisfying depends on the topic and purpose of the page. A High 
quality page on a broad topic with a lot of available information will have more content than a High quality page ona 
narrower topic. Here are some examples of pages with a satisfying amount of high quality MC. 


Siberian Husky The Siberian Husky (a breed of dog) is a narrow topic. Although this encyclopedia landing page has less MC 
Information Page than some encyclopedia pages on broader topics, it has a satisfying amount of clearly written, high quality MC. 


This shopping page on a reputable shopping website has a satisfying amount of high quality MC. The page 
provides the manufacturer’s product specs, as well as original product information, over 90 user reviews, 
shipping and returns information, multiple images of the product, etc. Note: Some of the MC is behind links on 
the page (“item details,” “item specifications,” “guest reviews,” etc.). Even though you have to click these links 
to see the content, it is still considered MC. 


Kitchen Stand Mixer 
Shopping Page 


This movie review written by a movie critic has a satisfying amount of high quality MC. Time, effort, and 


i i g . . og: . . . 
Movie Review Page talent/skill went into writing this movie review. 
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4.3 Clear and Satisfying Website Information: Who is Responsible and Customer Service 


Understanding who is responsible for a website is a critical part of assessing E-A-T for most types of websites. High 
quality pages should have clear information about the website so that users feel comfortable trusting the site. 


The amount of information needed for E-A-T assessment depends on the type of website. For example, YMYL websites 
demand a high degree of trust, so they generally need satisfying information about who is responsible for the content of 
the site. In addition, High quality stores and financial transaction websites also need clear and satisfying customer service 
information to help users resolve issues. 


Other websites that are not YMYL websites may need less website information, depending on the purpose of the website. 
For example, an email address may be sufficient for some non-YMYL websites. 


4.4 Positive Reputation 


Reputation is an important criteria when using the High rating, and informs the E-A-T of the page. While a page can merit 
the High rating with no reputation, the High rating cannot be used for any website that has a convincing negative 
reputation. Remember that when doing research, make sure to consider the reasons behind a negative rating and not 
just the rating itself. 


4.5 A High Level of Expertise/Authoritativeness/Trustworthiness (E-A-T) 


High quality pages and websites need enough expertise to be authoritative and trustworthy on their topic. Remember 
that there are “expert” websites of all types, even gossip websites, fashion websites, humor websites, forum and Q&A 
pages, etc. In fact, some types of information are found almost exclusively on forums and discussions, where a 
community of experts can provide valuable perspectives on specific topics. 


Think about the topic of the page. What kind of expertise is required for the page to achieve its purpose well? The 
standard for expertise depends on the topic of the page. For example, high quality news articles and information pages 
on scientific topics should represent established scientific consensus where such consensus exists. 


4.6 Examples of High Quality Pages 


Webpage’/Type of Content High Quality Characteristics PQ Rating and Explanation 


High: News 1 | | p= | 


Lowest Medium Highest 


A satisfying amount of high quality MC 


Homepage of a newspaper Positive reputation (website) 


(YMYL) This newspaper has won seven Pulitzer Prize 


awards. 


| | p= 


Lowest Medium Highest 


High Mews2 A satisfying amount of high quality MC 


Positive reputation (website) This is an article on a newspaper website that 

High E-A-T of the publisher and/or author | has won ten Pulitzer Prize awards. There is a 
satisfying amount of high quality MC. While 
there are Ads on the page, the MC is easy to 
find and the Ads are clearly labeled as Ads. 


Article on a newspaper website 
(YMYL) 
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Webpage’/Type of Content 


High: News 3 


Opinion article on a newspaper 
website 


High: Government agency 


US Naval Observatory Master 
Clock page 


High: Humor 


Article on a humor website 


High: Small business 1 


Local fish & chips restaurant 


High: Small business 2 


Local preservation center 


High: Blog post 


Parenting article about strollers 


High Quality Characteristics PQ Rating and Explanation 


A satisfying amount of high quality MC 
Positive reputation (website) 
High E-A-T of the publisher and/or author 


High E-A-T for the purpose of the page 


High quality humorous MC 
Positive reputation (website) 
Expertise as a farcical humor website 


High E-A-T for the purpose of the page 


High E-A-T for the purpose of the page 
A satisfying amount of high quality MC 


A satisfying amount of high quality MC 
Very positive reputation (website) 
Specifically, high E-A-T and positive 
reputation for this specific blog and author 
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Lowest Medium 


a 


Highest 


This is an opinion piece written by the Editorial 
Board of the the largest newspaper in the U.S. 
state of Minnesota. The page is clearly labelled 
as an opinion piece, welcoming a new football 
coach to the University of Minnesota. 


Lowest Medium 


a 


Highest 


The purpose of this page is to display the 
official US Naval Observatory Master Clock 
time in 7 different time zones. The page 
displays the clock information in a clear, 
easy-to-read format. The Naval Observatory is 
highly trustworthy and authoritative for this type 
of information. 


Lowest Medium 


n | 


Highest 


This website is well-known for its humorous, 
satirical articles. This is a cute example of a 
satisfying and funny article. 


| p | 


Lowest Medium Highest 


This is an “about us” page on a restaurant 
website. This page provides information on 
when the restaurant opened and what visitors 
can expect. Other pages on the website 
provide information about the restaurant 
including the address, menu, other contact 
information, etc. This website is highly 
authoritative because it is about itself. 


Lowest Medium 


p 


Highest 


This is the News and Updates section of a 
local preservation center selling poultry, 
vegetables, and more. 


Note: This example was added in 2014 so the 
“News and Updates” were timely. 


Lowest Medium 


— 


Highest 


This is a blog post on a newspaper that has 
won over 100 Pulitzer Prize awards. The 
author of this blog post has become known as 
an expert on parenting issues. She is a regular 
contributor to this and other media websites. 
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Webpage’/Type of Content 


High: Shopping 1 

e 
Backpack shopping page on a ° 
popular store website (YMYL) 


High: Shopping 2 


Bathroom décor page on a store 
website (YMYL) 


High: Shopping 3 


TomTom GPS page on an 
electronics website (YMYL) 


High: Video 1 


Saturday Night Live video on a 
video website 


High: Video 2 


“An Engineer’s Guide to Cats” 
video 


High: Game 


Online game 


A satisfying amount of high quality MC 
Positive reputation (website) 


High E-A-T for the purpose of the page 
Positive reputation (website) 


A satisfying or comprehensive amount of 
very high quality MC 

High E-A-T for the purpose of the page 
Positive reputation (website) 


A satisfying or comprehensive amount of 
very high quality MC 
High E-A-T for the purpose of the page 


A satisfying amount of high quality MC 
High E-A-T (everyday expertise) on the 
topic of cat ownership 


A satisfying or comprehensive amount of 
very high quality MC 

Positive reputation (website) as an 
educational tool for kids 
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High Quality Characteristics PQ Rating and Explanation 


Lowest Medium 


— | 


Highest 


The purpose of this page is to allow users to 
buy a school backpack. The page provides a 
lot of different backpack options, and some of 
them have user reviews. 


This is a well-known, reputable merchant, with 
detailed Customer Service information on the 
site. 


Lowest Medium 


p 


Highest 


This company sells its own line of high end, 
fashionable baby and children’s furniture and 
accessories. It has a positive reputation as well 
as expertise in these specific types of goods. 
Many products sold on the site are unique to 
this company. 


Lowest Medium 


p= 


Highest 


There is a very large quantity of MC on this 
page. Note that the tabs on the page lead to 
even more information, including many 
customer reviews. The tabs should be 
considered part of the MC. 


Lowest Medium 


p 


Highest 


The MC of this video page is an episode of 
Saturday Night Live, a very popular television 
show. 


Lowest Medium 


p 


Highest 


This is a humorous, high quality, well produced 
video with more than 6.5 million views (and 
counting), illustrating the proper care and 
practical benefits of cats. The two engineers in 
the video have everyday expertise on cat 
ownership. For a humorous video like this one, 
expertise does not have to be in the form of a 
trained veterinarian. 


Lowest Medium 


p 


Highest 


Highly engaging game with multiple levels that 
could entertain a child for hours. This website 
has a positive reputation for educating children 
about animals. 
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Webpage’/Type of Content 


High: Q&A 


How long do cancer patients live 


High: Encyclopedia 


Encyclopedia article about the 
American Civil War 


High: Forum 1 


Authenticating a luxury designer 
purse 


High: Forum 2 


Landscaping an aquarium 


High Quality Characteristics PQ Rating and Explanation 


e A satisfying amount of high quality MC in 
the form of personal narratives 


A satisfying or comprehensive amount of 
high quality MC 

Positive reputation (website) 

High E-A-T for the article 


e High E-A-T for the purpose of the page 
e A satisfying amount of high quality MC 


e High E-A-T for the purpose of the page 
e A satisfying amount of high quality MC 
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Lowest Medium 


— 


Highest 


The question on the page asks how long 
people live with cancer. There are many 
responses describing how long a loved one 
lived after diagnosis. There is very little 
medical advice and the focus of the page is 
sharing personal experience. Many responses 
are heartfelt and well written. 


Lowest Medium 


p 


Highest 


This is a detailed article about the American 
Civil War. The citations support the E-A-T of 
this article. 


Note: Although much of the content is visible on 
the mobile page, we consider the content under 
the headings (which you need to expand) to be 
part of the MC. (To see the screenshot, you will 
need to zoom in to the image.) 


Lowest Medium 


p 


Highest 


This forum is well-known for discussions on 
luxury designer purses. On this particular 
forum page, members are consulting forum 
experts who have expertise authenticating bags 
from this brand. These experts can tell if a 
particular bag is authentic or fake. While there 
is an ad at the top and a few ads within the 
forum message, it does not distract from the 
MC, which is easy to find. 


(To see the screenshot, you will need to zoom 
in to the image.) 


Lowest Medium 


p 


Highest 


This discussion focuses on the landscaping for 
a particular paludarium (an aquarium with 
terrestrial and aquatic elements). There is a 
lot of discussion and interaction between forum 
members about the types of materials and 
species used in the aquarium. The posts show 
expertise in a niche topic aquarium 
landscaping. 
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5.0 Highest Quality Pages 


Highest pages are very satisfying pages that achieve their purpose very well. The distinction between High and Highest 
is based on the quality of MC as well as the level of E-A-T and reputation of the website. 


What makes a page Highest quality? A Highest quality page may have the following characteristics: 


e Very high level of Expertise, highly Authoritative, and highly Trustworthy for the purpose of the page (E-A-T), 
including the E-A-T of the publisher and/or individual author for news articles and information pages on YMYL 
topics. 

e A satisfying amount of high quality MC. 

e Highly satisfying website information and/or information about who is responsible for the website or for stores and 
pages involving financial transactions, highly satisfying customer service reputation is very important. 

e Very positive website reputation for a website that is responsible for the MC on the page. 


5.1 Very High Quality MC 


We will consider the MC of the page to be very high or highest quality when it is created with a high degree of time and 
effort, and in particular, expertise, talent, and skill—this may provide evidence for the E-A-T of the page. Very high quality 
MC may be created by experts, hobbyists, or even people with everyday expertise. Our standards depend on the 
purpose of the page and the type of content. However, for news articles and information pages on YMYL topics, there is a 
high standard for accuracy and well-established medical/scientific/historical consensus where such consensus exists. 

The Highest rating may be justified for pages with a satisfying or comprehensive amount of very high quality MC. 


5.2 Very Positive Reputation 


Reputation research is important when giving Highest ratings, and is evidence of the E-A-T of the page. Very positive 
reputation is often based on prestigious awards or recommendations from known experts or professional societies on the 
topic of the page. Wikipedia and other informational sources can be a good starting point for reputation research. 


For some topics, such as humor or recipes, less formal expertise is OK. For these topics, popularity, user engagement, 
and user reviews can be considered evidence of reputation. For topics that need less formal expertise, websites can be 
considered to have a positive reputation if they are highly popular and well-loved for their topic or content type, and are 

focused on helping users. 


5.3 Very High Level of E-A-T 

Highest quality pages and websites have a very high level of expertise or are highly authoritative or highly trustworthy. 
Formal expertise is important for topics such as medical, financial, or legal advice. Expertise may be less formal for topics 
such as recipes or humor. An expert page on cooking may be a page on a professional chef's website, or it may be 
someone who posts popular cooking videos on YouTube. Please value life experience and “everyday expertise.” For 
some topics, the most expert sources of information are ordinary people sharing their life experiences on personal blogs, 


forums, reviews, discussions, etc. 


Think about what expertise, authoritativeness, and trustworthiness mean for the topic of the page. Who are the experts? 
What makes a source trustworthy for the topic? What makes a website highly authoritative for the topic? 
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5.4 


Webpage’/Type of Content 


Highest: News 


Article on a newspaper website 


Highest: Informational 1 


Fact-checking page on an 
Internet rumor debunking website 


Highest: Informational 2 


Ball gown wedding dress page 


Highest: Government agency 


Yosemite National Park 
information 


Examples of Highest Quality Pages 


Highest Quality Characteristics PQ Rating and Explanation 


Very high level of E-A-T for the purpose 
of the page, including of the publisher 
and/or authors for a news article 

A satisfying or comprehensive amount of 
very high quality MC for the purpose of 
the page 

Very positive reputation (website) 


A satisfying or comprehensive amount of 
very high quality MC 

Very high level of E-A-T for the purpose 
of the page 

Positive reputation (website) 


A satisfying or comprehensive amount of 
very high quality MC 

Very high level of E-A-T for the purpose 
of the page 

Positive reputation (website) 


Very high level of E-A-T for the purpose 
of the page 

A satisfying or comprehensive amount of 
very high quality MC for the purpose of 
the page 


Copyright 2017 


Lowest Medium Highest 


This is a feature article from a newspaper that 
has won 40 Pulitzer Prize awards and is the 
largest newspaper in the U.S. by circulation at 
the time this example was written. 


The page has a comprehensive amount of very 
high-quality MC, including in-depth reporting by 
two authors with extensive knowledge and 
experience in investigative journalism. 


| | p 


Lowest Medium Highest 


This page is on a well-known fact-checking and 
debunking website that covers urban legends, 
Internet rumors, and other stories of unknown 
or questionable origin. 


The page has a very satisfying amount of MC 
for users seeking to validate a claim about a 
documentary film about mermaids. Users can 
trust the information on this page due to the 
website's positive reputation and high level of 
expertise in debunking non-YMYL stories of 
this type. 


Lowest Medium 


= 


Highest 


This page has a very satisfying amount of MC 
for users interested in ball gown wedding 
dresses. An abundance of pictures, plus 
options to view by price range, style, etc., are 
part of what makes this page so satisfying. 
This page is on a popular wedding planning 
website. 


Lowest Medium Hignest 


This page has a highly satisfying amount of 
helpful information about Yosemite National 
Park, including traffic alerts and links to other 
parts of the website with additional information. 
Since the page is on the official national park 
website, it is uniquely authoritative. 
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Webpage’/Type of Content 


Highest: Technical 1 


Software tips 


Highest: Technical 2 


Linux Kernel archives 


Highest: Financial 1 


Credit report information website 
(YMYL) 


Highest: Financial 2 
Tax forms page (YMYL) 


Highest: Magazine Article 1 


Article titled “Secret Fears of the 
Super-Rich” 


Highest: Magazine Article 2 


Interview with musical artist 


Highest Quality Characteristics PQ Rating and Explanation 


Very high level of E-A-T for the purpose 
of the page 

A satisfying or comprehensive amount of 
high quality MC 


Very high level of E-A-T for the purpose 
of the page 

A satisfying or comprehensive amount of 
high quality MC 


Very high level of E-A-T for the purpose 
of the page 
Very positive reputation (website) 


Very high level of E-A-T for the purpose 
of the page 

A satisfying or comprehensive amount of 
very high quality MC for the purpose of 
the page 


Very high level of E-A-T for the purpose 
of the page 

A satisfying or comprehensive amount of 
very high quality MC 

Very positive reputation (website) 


Very high level of E-A-T for the purpose 
of the page 

A satisfying or comprehensive amount of 
very high quality MC 

Very positive reputation (website) 
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Lowest Medium Hignest 


This page offers a satisfying amount of helpful 
and very high quality MC for the purpose of the 
page. Because the page is on the official 
website of the company that produces the 
software, it is highly authoritative. 


Lowest Medium Highest 


From this Wikipedia article, we learn that 
“Kernel.org is a main repository of source code 
for the Linux kernel, the base of the popular 
Linux operating system. It makes all versions 
of the source code available to all users... The 
main purpose of the site is to host a repository 
for Linux kernel developers and maintainers of 
Linux distributions.” 


This website is the most authoritative source for 
information about Linux. 


Lowest Medium Highest 


Users in the U.S. can obtain free credit reports 
on this website by providing their Social 
Security Number. This Wikipedia article tells us 
that this website is “the only federally mandated 
and authorized source for obtaining a free 
credit report.” 


Lowest Medium Highest 


The purpose of the page is to provide income 
tax forms and publications. The website 
represents an agency of the U.S. federal 
government that handles taxes, so the page 
and forms are uniquely authoritative and 
trustworthy. 


| | | 


Lowest Medium 


= 


Highest 


This is a very high quality and in-depth article 
on an award winning magazine website. 


Lowest Medium 


p 


Highest 


This magazine is very well-known and well 
regarded for content about artists and bands, 
and specifically has a positive reputation for 
interviews with musicians. There is a large 
quantity of high quality MC. 


26 


Webpage’/Type of Content 


Highest: Medical 1 


BMI calculator (YMYL) 


Highest: Medical 2 


Page about meningitis (YMYL) 


Highest: Medical 3 


Page about seasonal flu (YMYL) 


Highest: Medical 4 


Health information on a hospital 
page (YMYL) 


Highest: Entertainment 


Humor website 


Very high level of E-A-T for the purpose 
of the page 
Very positive reputation (website) 


Very high level of E-A-T for the purpose 
of the page 
Very positive reputation (website) 


Very high level of E-A-T for the purpose 
of the page 
Very positive reputation (website) 


Very high level of E-A-T for the purpose 
of the page 


A satisfying or comprehensive amount of 
very high quality MC 
Very positive reputation (website) 
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Highest Quality Characteristics PQ Rating and Explanation 


Lowest Medium Hignest 


The purpose of this page is to provide an online 
BMI (Body Mass Index) calculator. The 
calculator is functional and easy to use. 


The website represents an institute that is part 
of the National Institutes of Health. It has an 
extremely good reputation and is an expert in 
medical topics. 


Lowest Medium Highest 


This is a meningitis reference page ona 
trustworthy and authoritative website for a 
nonprofit medical research group. This website 
has a reputation of being one of the best web 
resources for medical information. 


Lowest Medium Highest 


This is an influenza reference page ona 
trustworthy and authoritative medical website. 
This website has a reputation of being one of 
the best web resources for medical information 
of this type. 


Lowest Medium Highest 


According to this Wikipedia article, this hospital 
is “currently regarded as one of the top 4 
hospitals in the United States as rated by U.S. 
News & World Report.” Users can trust 
medical information on this website. 


Lowest Medium Hignest 


This comedian has won multiple prestigious 
awards for his TV show (on the air when the 
example was written), and this is the homepage 
of his official website. 


Keep in mind that for any type of page, 
including humor websites, there may be a 
range of highest quality to lowest quality pages. 
Therefore, it’s important to evaluate the page 
using the same criteria as all other pages, 
regardless of what type of page it is. 
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Webpage’/Type of Content 


Highest: Shopping 


Backpack shopping page 
(YMYL) 


Highest: Login 


Online banking login page 
(YMYL) 


Highest: Charity 


Homepage of a charity (YMYL) 


Highest: Page with Error 
Message 


“Custom 404” page 


Highest: PDF File 


Campus map 


Highest Quality Characteristics PQ Rating and Explanation 


Very high level of E-A-T for the purpose 
of the page 

Very positive reputation (website) 

A satisfying or comprehensive amount of 
high quality MC 


A satisfying amount of high quality MC 
for its purpose (login page) 

Positive reputation (website) 

Very high level of E-A-T for the purpose 
of the page 


Very high level of E-A-T for the purpose 
of the page 
Very positive reputation (website) 


A satisfying amount of high quality MC 
for its purpose (error page) 

Very positive reputation (website) 

Very high level of E-A-T for the purpose 
of the page 


A satisfying or comprehensive amount of 
very high quality MC 

Very high level of E-A-T for the purpose 
of the page 
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| | p 


Lowest Medium Highest 


The purpose of this page is to provide 
information about, and allow users to buy, a 
specific type of school backpack. The page 
provides a lot of helpful product information, as 
well as 600 user reviews. 


Since the store produces this backpack, they 
are experts on the product, making the page on 
their own website authoritative. In addition, this 
store has a reputation for producing one of the 
highest quality and most popular school 
backpacks on the market. 


| | p 


Lowest Medium Highest 


This page has login functionality, as well as 
clear information about what the user is logging 
into. This is a large, popular bank with a good 
reputation. 


| | p 


Lowest Medium Highest 


This is a highly reputable charity according to 
multiple charity rating organizations. 


| | p 


Lowest Medium Highest 


This is an example of a “custom 404” page. 
These pages are designed to alert users that 
the URL they are trying to visit no longer exists. 
The MC of this page is the cartoon, the caption, 
and the search functionality, which is specific to 
the content of the website. Itis clear that time, 
effort, and talent were involved in its creation. 


This publication has a very positive reputation 
and is specifically known for its cartoons. 


Keep in mind that for any type of page, 
including pages with error messages, there 
may be a range of highest quality to lowest 
quality. Therefore, it’s important to evaluate the 
page using the same criteria as all other pages, 
regardless of what type of page it is. 


| | p 


Lowest Medium Highest 


This PDF file is a detailed campus map of a 
major university, which is hosted on the official 
university website. This is a highly authoritative 
source for this information. The map includes a 
listing of the all the buildings, parking 
structures, parking lots, construction areas, etc. 
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Webpage’/Type of Content 


Highest: Recipes 1 


Pumpkin pie recipe 


Highest: Recipes 2 


Chocolate chip cookie recipe on a 
recipe blog 


Highest: Video 1 


The band OK Go’s music video 
for the song “This Too Shall Pass” 


Highest: Video 2 


“Henri 2, Paw de Deux” video 


Highest: Video 3 


Saturday Night Live video on the 
TV network’s website 


Highest: Q&A 


Q&A page on abbreviations 


Highest Quality Characteristics PQ Rating and Explanation 


Very high level of E-A-T for the purpose 
of the page 

A satisfying amount of high quality MC 
and features 


Very high level of E-A-T for the purpose 
of the page 

A satisfying amount of high quality MC 
and features 


A satisfying or comprehensive amount of 
very high quality MC 

Very high level of E-A-T for the purpose 
of the page 


A satisfying or comprehensive amount of 
very high quality MC 

Very positive reputation for the content of 
this particular video 


A satisfying or comprehensive amount of 
very high quality MC 

Very high level of E-A-T for the purpose 
of the page 


Very high level of E-A-T for the purpose 
of the page 
Very positive reputation (website) 
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Lowest Medium 


== 


Highest 


The website is one of the most popular recipe 
websites and the author of the recipe is a 
well-known celebrity chef who specializes in 
southern cooking. 


This page has a large number of user reviews 
of the recipe, detailed preparation time 
information, a “recipe box” feature, etc. 


| | p 


Lowest Medium Highest 


Even though this user does not seem to be a 
well-known professional chef, recipes are an 
example of everyday expertise. The author of 
this blog has documented her extensive 
experimentation with a chocolate chip cookie 
recipe, and her expertise is demonstrated in the 
large quantity of high or highest quality MC. 


| | p 


Lowest Medium Highest 


This is a high quality, professionally produced 
video on the band’s official channel. 


| | p 


Lowest Medium Highest 


This is a high quality, popular video. It won the 
“Golden Kitty Award” for “Best Cat Video On 
The Internet” at the Walker Art Center's Internet 
Cat Video Film Festival. 


Lowest Medium 


=j 


Highest 


The MC of this video page is an episode of 
Saturday Night Live on the TV network’s official 
website, making it highly authoritative and 
trustworthy. Below the main video, there are 
many other videos that users may be interested 


Medium 


= 


Lowest Highest 


The purpose of the page is to answer questions 
about the usage of abbreviations. This website 
has a very good reputation as a reference for 
information on writing, publishing, etc. Itis 
considered highly authoritative and trustworthy 
for the topic of the page. 
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6.0 Low Quality Pages 


Low quality pages are unsatisfying or lacking in some element that prevents them from achieving their purpose well. 
These pages lack expertise or are not very trustworthy/authoritative for the purpose of the page. 


If a page has one of the following characteristics, the Low rating is usually appropriate: 


e The author of the page or website does not have enough expertise for the topic of the page and/or the website is 
not trustworthy or authoritative for the topic. In other words, the page/website is lacking E-A-T. 

The quality of the MC is low. 

There is an unsatisfying amount of MC for the purpose of the page. 

MC is present, but difficult to use due to distracting/disruptive/misleading Ads, other content/features, etc. 

There is an unsatisfying amount of website information for the purpose of the website (no good reason for 
anonymity). 

e The website has a negative reputation. 


6.1 Low Quality Main Content 


One of the most important criteria in PQ rating is the quality of the MC, which is determined by how much time, effort, 
expertise, and talent/skill have gone into the creation of the page, and also informs the E-A-T of the page. 


Consider this example: Most students have to write papers for high school or college. Many students take shortcuts to 
save time and effort by doing one or more of the following: 


e Buying papers online or getting someone else to write for them. 

e Including inaccurate information, such as making things up, stretching the truth, or creating a false sense of doubt 

about well-established facts. 

Writing quickly with no drafts or editing. 

Failing to cite sources, or making up sources where none exist. 

Filling the report with large pictures or other distracting content. 

Copying the entire report from an encyclopedia, or paraphrasing content by changing words or sentence structure 

here and there. 

e Using commonly known facts, for example, “Argentina is a country. People live there. Argentina has borders.” 

e Using a lot of words to communicate only basic ideas or facts, for example, “Pandas eat bamboo. Pandas eat a 
lot of bamboo. Bamboo is the best food for a Panda bear.” 


Unfortunately, the content of some webpages is similarly created. We will consider content to be Low quality if it is 
created without adequate time, effort, expertise, or talent/skill. Inaccurate or misleading information presented as fact is 
also a reason for Low or even Lowest quality ratings. Pages with low quality MC do not achieve their purpose well. 
Keep in mind that we have very different standards for pages on large, professionally-produced business websites than 
we have for small amateur, hobbyist, or personal websites. The quality of MC we expect for a large online store is very 
different than what we might expect for a small local business website. 

All PQ rating should be done in the context of the purpose of the page and the type of website. 


Important: Low quality MC is a sufficient reason to give a page a Low quality rating. 


6.2 Unsatisfying Amount of Main Content 


Some Low quality pages are unsatisfying because they have a small amount of MC for the purpose of the page. For 
example, imagine an encyclopedia article on a very broad topic such as World War II that has just a few paragraphs. 


Important: An unsatisfying amount of MC is a sufficient reason to give a page a Low quality rating. 
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6.3 Distracting/Disruptive/Misleading Titles, Ads, and Supplementary Content 


Some Low quality pages have adequate MC present, but it is difficult to use the MC due to disruptive, highly distracting, 
or misleading Ads/SC. Misleading titles can result in a very poor user experience when users Click a link only to find that 
the page does not match their expectations. 


6.3.1 Ads or SC that disrupt the usage of MC 


While we expect Ads and SC to be visible, some Ads, SC or interstitial pages (i.e., pages displayed before or after the 
content you are expecting) make it extremely difficult to use the MC. Pages that disrupt the use of the MC should be 
given a Low rating. A single pop-over Ad with a clear close button is not terribly disruptive, though may not be a great 
user experience. Here are two examples of situations we consider to be disruptive: 


e Ads that actively float over the MC as you scroll down the page and are difficult to close. It can be very hard to 
use MC when it is actively covered by moving, difficult-to-close Ads. 
e Aninterstitial page that redirects the user away from the MC without offering a path back to the MC. 


6.3.2 Prominent presence of distracting SC or Ads 


Users come to webpages to use the MC. Helpful SC and Ads can be part of a positive user experience, but distracting 
SC and Ads make it difficult for users to focus on and use the MC. Here are some examples of prominent and distracting 
SC or Ads: 


e Some webpages are designed to encourage users to click on SC that is not helpful for the purpose of the page. 
This type of SC is often distracting or prominently placed in order to lure users to highly monetized pages. Here is 
an example of a page with highly distracting SC in the right-hand column under the label “Top Posts & Pages." 
Here and here are more examples of pages with some very prominent and distracting SC images and/or Ads. 

e Either porn SC or Ads containing porn on non-Porn pages can be very distracting or even upsetting to users. 
Please refresh the page a few times to see the range of Ads that appear, and use your knowledge of the locale 
and cultural sensitivities to make your rating. For example, an ad for a model in a revealing bikini is probably 
acceptable on a site that sells bathing suits. However, an extremely graphic porn ad may warrant a Low (or even 
Lowest) rating. 


6.3.3 Misleading Titles, Ads, or SC 


It should be clear what parts of the page are MC, SC, and Ads. It should also be clear what will happen when users 
interact with content and links on the webpage. If users are misled into clicking on Ads or SC, or if clicks on Ads or SC 
leave users feeling surprised, tricked or confused, a Low rating is justified. 


Here are some examples of misleading Titles, Ads or SC: 


e Atfirst glance, the Ads or SC appear to be MC. Some users may interact with Ads or SC, believing that the Ads 
or SC is the MC. Here is an example. 

e Ads appear to be SC (links) where the user would expect that clicking the link will take them to another page 
within the same website, but actually take them to a different website. Some users may feel surprised or 
confused when clicking SC or links that go to a page on a completely different website. 

e Ads or SC that entice users to click with shocking or exaggerated titles, images, and/or text. These can leave 
users feeling disappointed or annoyed when they click and see the actual and far less interesting content. Here is 
an example of a page with shocking/exaggerated titles, images, and text in the Ads and SC. 

e Titles of pages or links/text in the SC that are misleading or exaggerated compared to the actual content of the 
page. This can result in a very poor user experience when users read the title or click a link only to find that the 
page does not match their expectations. Here is an example of a page with a misleading/exaggerated title. 


Summary: The Low rating should be used for disruptive or highly distracting Ads and SC. Misleading Titles, Ads, or SC 


may also justify a Low rating. Use your judgment when evaluating pages. User expectations will differ based on the 
purpose of the page and cultural norms. 
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6.4 Negative Reputation 


Reputation research is required for all PQ rating tasks unless you have previously researched the reputation of the 
website. Extremely negative, malicious, or financially fraudulent reputation information should result in a Lowest rating. 
Credible negative (though not malicious or financially fraudulent) reputation is a reason for a Low rating, especially for a 
YMYL page. 


Please exercise care when researching the reputation of businesses. Most businesses have some negative reviews, 
especially for customer service. Try to find as many reviews and ratings as possible and read the details of negative 
reviews and low ratings before inferring that the business has a negative reputation. 


Reputation research is also critical for information pages and news websites, particularly those on YMYL topics. The lack 
of any reputation information at all for a YMYL website may be a reason for a Low rating. 


Important: Negative reputation is sufficient reason to give a page a Low quality rating. Evidence of truly malicious or 
fraudulent behavior warrants the Lowest rating. 


6.5 Lacking Expertise, Authoritativeness, or Trustworthiness (E-A-T) 


Some topics demand expertise for the content to be considered trustworthy. YMYL topics such as medical advice, legal 
advice, financial advice, etc. should come from authoritative sources in those fields, must be factually accurate, and must 
represent scientific/medical consensus within those fields where such consensus exists. Even everyday topics, such as 
recipes and house cleaning, should come from those with experience and everyday expertise in order for the page to be 
trustworthy. 


You should consider who is responsible for the content of the website or content of the page you are evaluating. Does the 
person or organization have sufficient expertise for the topic? If expertise, authoritativeness, or trustworthiness is lacking, 
use the Low rating. 


User-generated websites span the Page Quality rating spectrum. Note that in some cases, contributors choose their own 
topics with no oversight and may have very poor writing skills or no expertise in the topic of the page. Contributors may 
be paid per article or word, and may even be eligible for bonuses based on the traffic to their pages. Depending on the 
topic, pages on these websites may not be trustworthy. 


Important: Lacking appropriate E-A-T is sufficient reason to give a page a Low quality rating. 
6.5.1 Unsatisfying Amount of Information about the Website 


We expect some form of website information for many or most websites. However, the amount of website information 
needed depends on the purpose of the website. For personal websites, an email address alone may be sufficient. 


Stores and websites that process financial transactions require a high level of user trust. If a store or financial transaction 
website has just an email address and physical address, it may be difficult to get help if there are issues with the 
transaction. Likewise, many other types of YMYL websites also require a high degree of user trust. 


Important: For YMYL pages and other pages that require a high level of user trust, an unsatisfying amount of any of the 


following is a reason to give a page a Low quality rating: customer service information, contact information, or information 
about who is responsible for the website. For other types of websites, use your judgment. 
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6.6 Examples of Low Quality Pages 


Type of Webpage/Content Low Quality Characteristics PQ Rating and Explanation 


Lowest Medium Highest 


The writing of this article is unprofessional, 

e Low quality MC including many grammar and punctuation 

e Lacking E-A-T errors. The MC also appears to have been 
paraphrased from a science article found on a 
different source, but with factual inaccuracies 
introduced throughout. The combination of low 
quality MC and lack of E-A-T justify the 
Lowest+ to Low rating. 


Lowest Medium H ig hest 


Low: News 1 


Article about nuclear power 


The title of the article is misleading and does 
not reflect the actual content of the page. The 
Low: News 2 MC does not explain how to get rid of toxins or 


: Tia a title what the word "toxins" in the headline refers to. 


Article titled "Getting Rid Of e Lacking E-A-T 


Toxins After The Holiday Season" The content also has many problems: the 
writing quality is poor, and the article includes 
meaningless statements such as "water 
therapy is one of the easiest ways of beauty 
regimen since it will give enough moisture on 
the skin". The article fails to cite sources, and 
there is no evidence of E-A-T. 


Lowest Medium Highest 


Low: Informational 1 


Low quality MC This page and website have many of 

, ; Disruptive Ads characteristics of Low quality pages. Close 
SO pinhoay parny [ood ideas observation shows MC that contains mostly 
commonly known information and poor quality 
writing. The MC is broken up by large Ads that 
disrupt the user experience. 


| | | 
Lowest Medium Highest 


Low: Informational 2 Unsatisfying amount of MC for the 


purpose of the page Some websites rely on users to create virtually 
all of their MC. In this case, the MC is the 
user’s question. If there are no answers, the 
amount of MC on the page is unsatisfying. 


Q&A unanswered question 
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Type of Webpage/Content 


Low: Article 1 
Low or Lowest quality MC 
Article with tips for dressing for Lacking E-A-T 


the office 


LC ANGER Low or Lowest quality MC 


Using ginger for your health Lacking E-A-T 


Ss e Low quality MC: commonly known 


information only 


How to adopt children from Iraq e Lacking E-A-T on a YMYL topic 


(YMYL) 


Low: Financial 


Page with advice on picking a e Lacking E-A-T on a YMYL topic 
quality stock for investment 
(YMYL) 


Low: Medical 
Page with information about how e ‘Lacking EAT ona 'YMYLtopic 
long the flu lasts (YMYL) 
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Low Quality Characteristics PQ Rating and Explanation 


p=) | | 
Lowest Medium Highest 


This content has many problems: 
inaccurate/meaningless information and 
complete lack of editing with poor spelling and 
grammar—both of these characteristics in 
combination justify the Lowest+ to Low rating. 


p=) | | 
Lowest Medium Highest 


This content has many problems: garbled 
information and a lack of editing. Both of these 
characteristics in combination justify the 
Lowest+ to Low rating. In addition, lacking 
E-A-T on this topic is concerning. 


Lowest Medium Highest 


This is a YMYL topic. However, the steps listed 
here are merely commonly-known information 
that would be of little benefit to someone 
interested in adopting a child from Iraq. 


For example, step 1 says “Choose an adoption 
agency” and suggests looking in a phone book. 
There is almost no information specific to Iraq. 
There is no evidence of expertise on adoption 
from the description about the author. 


Note that some Ads have the same format as 
links to other articles on this website, potentially 
making it a bit harder for users to visually 
distinguish these Ads from the MC and SC. 


| | | 


Lowest Medium Highest 


There is no evidence that the author has 
financial expertise. Because this is a YMYL 
financial article, lacking expertise is a reason 
for a Low rating. 


Lowest Medium Highest 


There is no evidence that the author has 
medical expertise. Because this is a YMYL 
medical article, lacking expertise is a reason for 
a Low rating. 
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Type of Webpage/Content Low Quality Characteristics PQ Rating and Explanation 


a} | | 
Lowest Medium Highest 


This page and website have many of 
characteristics of Low quality pages. The MC is 
low quality and lacks important information. 
For example, it gives no indication of how to 
e Low quality and unsatisfying amount of make a crust and doesn’t list premade crust as 
MC an ingredient. It would be very difficult to 
actually make an apple pie using this recipe. 


Low: Recipes 1 


Apple pie recipe 


This page is lacking the kind of helpful content 
we expect in recipe pages, such as user 
reviews or comments. Without such 
information, it’s hard to tell if the recipe is any 
good. The small pink text at the top is not 
helpful for users. 


==) | | 
Lowest Medium Highest 
; ip : : - 
Low. Recipes 2 Very distracting SC or Ads (not clear) in 
the right column 
Disruptive Ads in the middle column 


This is an example of distracting and disruptive 
Ads/SC: there are large Ads in the main column 
pushing down the MC as well as highly 
distracting content on the right that are labeled 
“Top Posts & Pages.” It is unclear whether 
these are SC or Ads. 


Gluten-free New York cheesecake 
recipe 


I | | 


Lowest Medium Highest 


Low: Q&A1 


Lacking E-A-T There are 94 answers to this question with a 
few results that seem helpful. Many of the 
posts appear to be deliberately incorrect or 
misleading, including the top answer, which is 
labeled the “best answer.” 


Page about Native American 
customs 


| | | 
Lowest Medium Highest 


Please read the MC (areas with red boxes 
around it), including the completely unhelpful 
"answer." This answer is so unhelpful, we can 
Low: Q&A 2 e Misleading Ads consider this question to be unanswered. This 
e Unsatisfying amount of MC for the page has an unsatisfying amount of MC. 
Page about a 2002 Volvo part purpose of the page 
In addition to a very unhelpful “answer,” the 
page design makes it difficult to distinguish the 
MC from Ads. For example, below the answer, 
we see a "sponsored answer," which has the 
same format as the real answer, but is actually 
an Ad and not an answer to the question—this 
is misleading to users. 
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Type of Webpage/Content Low Quality Characteristics PQ Rating and Explanation 


Lowest Medium Highest 


This is a page from a Q&A site with an 
unanswered question. Q&A pages exist to 
answer user questions. Pages with 


Low: Q&A 3 e Misleading Ads/SC unanswered questions should generally be 


e  Unsatisfying amount of MC for the rated Low because they have little MC (just a 


Page about water and soil purpose of the page question anano answerang:donitachieve 
their purpose well. 


In addition to having no answer, this page has 
Ads and links to other questions (misleadingly 
labeled as “Relevant answers”) displayed 
prominently, which users may mistake for 
answers to the question. It takes a moment to 
notice that this page actually has no answer. 
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7.0 Lowest Quality Pages 


This section describes Lowest quality pages. The examples at the end in Section 7.10 are critical for understanding 
these concepts, so please review them carefully. 


Remember that websites and pages should be created to be helpful for users. 


Important: Websites and pages that are created to harm users, mislead or misinform users, or only make money with no 
attempt to help users should be rated Lowest. Webpages created with the sole purpose of promoting hate or violence 
against a group of people based on criteria including (but not limited to) race or ethnicity, religion, gender, nationality or 
citizenship, disability, age, sexual orientation, or veteran status should also be rated Lowest. 


Here are some types of pages or websites that are highly untrustworthy, unreliable, unauthoritative, inaccurate or 
misleading—these pages should always receive the Lowest rating: 


e Harmful or malicious pages or websites. 

e True lack of purpose pages or websites. 

e Deceptive pages or websites, such as: 

o Pages or websites that misrepresent the website's owner or purpose, such as by impersonating a 
different site (e.g., copied logo or branding of an unaffiliated site, URL that mimics another site's name). 

o Pages or websites that appear to be deliberate attempts to misinform or deceive users by presenting 
factually inaccurate content (e.g., fake product reviews, demonstrably inaccurate news, etc.). 

Pages or websites created to make money with little to no attempt to help users. 

Pages with extremely low or lowest quality MC. 

Pages on YMYL websites that are so lacking in website information that it feels untrustworthy. 

Pages or websites created with no expertise or pages that are highly untrustworthy, unreliable, unauthoritative, 

inaccurate, or misleading. 

e Pages or websites with factually inaccurate content that can cause harm to users, regardless of their purpose or 
intent (e.g., inaccurate medical information; inaccurate information about how to vote, obtain a visa or green card, 
pay government taxes, etc.). 

e Hacked, defaced, or spammed pages. 

e Websites that have extremely negative or malicious reputations. 

e Violations of the Google Webmaster Quality Guidelines, such as pages on websites with little or no original 
content on the website as a whole. 

e Pages or websites created with the sole purpose of promoting hate or violence against a group of people based 
on criteria including (but not limited to) race or ethnicity, religion, gender, nationality or citizenship, disability, age, 
sexual orientation, or veteran status. 


Finally, Lowest+ may be used both for pages with many/all low quality characteristics. Lowest+ may also be used for 
pages whose lack of a single Page Quality characteristic makes you question the true purpose of the page. 


7.1 Harmful or Malicious Pages 


Some pages are harmful or malicious. For example, pages designed to "phish" for the user’s government-issued 
identification number (such as a Social Security Number in the United States; other examples here), bank account 
information, or credit card information are harmful because the purpose is to steal private information. Malicious 
download pages are another type of harmful page. 


Harmful or malicious pages should be rated Lowest. 


7.2 Lack of Purpose Pages 


Sometimes it is impossible to figure out the purpose of the page. Such pages serve no real purpose for users. For 
example, some pages are deliberately created with gibberish or meaningless (nonsense) text. 


No matter how they are created, true lack of purpose pages should be rated Lowest quality. 
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12 Deceptive Pages 


We will consider a page to be “deceptive” if it is designed to deceive users or trick search engines, rather than to help 
users. The following sections in 7.3 describe several types of deceptive pages. Note that some pages may use a variety 
of these techniques. All deceptive pages should be rated Lowest. 


For more information on deceptive pages, see the Google Webmaster Quality Guidelines. 
7.3.1 Deceptive Page Purpose 


Deceptive webpages appear to have a helpful purpose (the stated purpose), but are actually created for some other 
reason. Use the Lowest rating if a webpage page is deliberately created to deceive and potentially harm users in order to 
benefit the website. 


Here are some examples of Lowest websites or webpages with deceptive page purpose: 


e Awebsite pretends to be or mimics the look of a well-known store, bank, online service, or social network, but 
instead exists to “phish” or steal passwords or personal information (for example, government ID number, bank 
account information, personal medical data, etc.). We also consider this type of website to have a malicious 
purpose. 

e A webpage claims to be a survey, perhaps offering money or a prize, but instead exists to steal passwords or 
personal information (for example, government ID number, bank account information, personal medical data, 
etc.). We also consider this type of website to have a malicious purpose. 

e A webpage or website that impersonates a different site (e.g., copied logo or branding of an unaffiliated site, URL 
that mimics another site's name, etc.). 

e A webpage or website looks like a news source or information page, but in fact has articles with factually 
inaccurate information to manipulate users in order to benefit a person, business, government, or other 
organization politically, monetarily, or otherwise. 

e A non-satirical webpage or website presents unsubstantiated conspiracy theories or hoaxes as if the information 
were factual. 

e A webpage claims to offer an independent review or share other information about a product, but is in fact created 
to make money for the owner of the website without attempting to help users. For example, the MC may contain 
intentionally misleading or inaccurate information created with the sole purpose of getting users to click on 
monetized links or buy the product. 

e A website claims to be the personal website of a celebrity, but the website is actually created to make money for 
the owner of the website without the permission of the celebrity. For example, the page may have false 
testimonials for a product and is created for the sole purpose of getting users to click on monetized links or buy 
the product. 


Sometimes it is difficult to determine the real purpose of a page. Look carefully and use your judgment. If you believe the 
webpage or website was created deliberately to deceive users for the benefit of the website, and with the potential to 
cause harm to the user, use the Lowest rating. 


7.3.2 Deceptive Page Design 


Some pages are deliberately designed to manipulate users to take an action that will benefit the owner of the website 
rather than help the user. 


We consider the following kinds of pages to be deceptive webpages because users did not get what they expected. Use 
the Lowest rating if the page is deliberately designed to manipulate users with little or no effort to provide helpful MC. 
Here are some common types of deceptive pages: 


e Pages that disguise Ads as MC. Actual MC may be minimal or created to encourage users to click on the Ads. 
For example, fake search pages (example) that have a list of links that look like a page of search results. If you 
click on a few of the links, you will see that the page is just a collection of Ads disguised as search engine results. 
A “search box” is present, but submitting a new query just gives you a different page of Ads disguised as search 
results. 

e Pages that disguise Ads as website navigation links. For example, fake directory pages (example) that look 
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like a personally curated set of helpful links, possibly with unique descriptions. In reality, the links are Ads or links 
to other similar pages on the site. Sometimes the descriptions of the links are unrelated to the page. 

e Pages where the MC is not usable or visible. For example, a page that has such a large amount of Ads at the 
top of the page (before the MC), so that most users will not see the MC, or a page where the MC is invisible text. 


Take a good look at the page and use your judgment. If you believe the page was deliberately created to manipulate 
users to click on Ads, monetized links, questionable download links, etc., rather than help users, the page should be rated 
Lowest. 


7.4 Lowest Quality Main Content 


In this guideline, we'll judge the quality of the MC by thinking about how much time, effort, expertise, and talent/skill was 
involved in content creation. 


If very little or no time, effort, expertise, or talent/skill has gone into creating the MC, use the Lowest quality rating. Ifthe 
content is misleading or inaccurate in a way that may harm or deceive users, also use the Lowest quality rating. 


All of the following should be considered either lowest quality MC or no MC: 
e No helpful MC at all or so little MC that the page effectively has no MC. 
e MC that consists almost entirely of “keyword stuffing.” 
e Gibberish or meaningless MC. 
e "Auto-generated” MC, or MC that was otherwise created with little to no time, effort, expertise, manual curation, or 
added value for users. 
Misleading or inaccurate informational content about YMYL topics. 
e Pages or websites that appear to be deliberate attempts to misinform or deceive users by presenting factually 
inaccurate content. 
e Pages or websites with factually inaccurate content that may harm or deceive users, regardless of their purpose 
or intent. 
e MC that consists almost entirely of content copied from another source with little time, effort, expertise, manual 
curation, or added value for users. 


Pages with lowest quality MC should be rated Lowest. 
7.4.1 No Main Content 


If a page is deliberately created with no MC, use the Lowest rating. Why would a page exist without MC? Pages with no 
MC are usually lack of purpose pages or deceptive pages. 


Webpages that are deliberately created with a bare minimum of MC, or with MC that is completely unhelpful for the 
purpose of the page, should be considered to have no MC. 


Pages deliberately created with no MC should be rated Lowest. 

7.4.2 “Keyword Stuffed” Main Content 

Pages may be created to lure search engines and users by repeating keywords over and over again, sometimes in 
unnatural and unhelpful ways. Such pages are created using words likely to be contained in queries issued by users. 


Keyword stuffing can range from mildly annoying to users, to complete gibberish. 


Pages created with the intent of luring search engines and users, rather than providing meaningful MC to help users, 
should be rated Lowest. 
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7.4.3 Automatically-Generated Main Content 


Entire websites may be created by designing a basic template from which hundreds or thousands of pages are created, 
sometimes using content from freely available sources (such as an RSS feed or API). These pages are created with no 
or very little time, effort, or expertise, and also have no editing or manual curation. 


Pages and websites made up of auto-generated content with no editing or manual curation, and no original content or 
value added for users, should be rated Lowest. 


7.4.4 Misleading or Inaccurate YMYL Informational Main Content 


Misleading or inaccurate MC on an informational page should cause the overall PQ rating to be Low or Lowest. If the 
inaccurate information could potentially harm users in any way (e.g., inaccurate medical information; inaccurate 
information about how to vote, obtain a visa or green card, pay government taxes, etc.), use the Lowest rating. If the 
inaccuracy appears to be deliberate and intended to mislead (e.g., fake product reviews, demonstrably inaccurate news, 
etc.), use Lowest. Please use your judgment. 


7.4.5 Copied Main Content 
Every page needs MC. One way to create MC with no time, effort, or expertise is to copy it from another source. 


Important: We do not consider legitimately licensed or syndicated content to be “copied” (see here for more on web 
syndication). Examples of syndicated content in the U.S. include news articles by AP or Reuters. 


The word “copied” refers to the practice of “scraping” content, or copying content from other non-affiliated websites without 
adding any original content or value to users (see here for more information on copied or scraped content). 


If all or most of the MC on the page is copied, think about the purpose of the page. Why does the page exist? What value 
does the page have for users? Why should users look at the page with copied content instead of the original source? 


Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no time, effort, 
expertise, manual curation, or added value for users. Such pages should be rated Lowest, even if the page assigns 
credit for the content to another source. 


7.4.6 More About Copied Content 
All of the following are considered copied content: 


e Content copied exactly from an identifiable source. Sometimes an entire page is copied, and sometimes just 
parts of the page are copied. Sometimes multiple pages are copied and then pasted together into a single page. 
Text that has been copied exactly is usually the easiest type of copied content to identify. 


e Content that is copied, but changed slightly from the original. This type of copying makes it difficult to find 
the exact matching original source. Sometimes just a few words are changed, or whole sentences are changed, 
or a “find and replace” modification is made, where one word is replaced with another throughout the text. These 
types of changes are deliberately done to make it difficult to find the original source of the content. We call this 
kind of content “copied with minimal alteration.” 


e Content copied from a changing source, such as a search results page or news feed. You often will not be 
able to find an exact matching original source if it is a copy of “dynamic” content (content that changes frequently). 
However, we will still consider this to be copied content. 


Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no time, effort, 


expertise, manual curation, or added value for users. Such pages should be rated Lowest, even if the page assigns 
credit for the content to another source. 
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7.4.7 How to Determine if Content is Copied 


How do you determine whether all or most of the MC is copied? How do you identify the original source of the content? 
These things can be difficult to determine, but the following steps may help. 


1. Copy a sentence or phrase in the text. It may be necessary to try a few sentences or phrases from the page just 
to be sure. When deciding what sentence or phrase to copy, try to find a sentence or series of several words 
without punctuation, unusual characters, or suspicious words that may have replaced the original text. 

2. Search on Google by pasting the sentence or phrase (surrounded by quotation marks) inside the Google search 
box. Try a few sentences from the page. 


Google 


"Dorothy uses the Golden Cap to summon the Winged monkeys" 


= 


Google Search I'm Feeling Lucky 


Here are some sentences and phrases from this page about “The Wizard of Oz’ film and how to search for 
them. 


Sentence or Phrase From a Sentence Query to Find the Sentence or Phrase 


Try the query both with and without quotation marks around 
A sentence on the page: the sentence: 

[ “All are convinced by Dorothy that the Wizard can help 
All are convinced by Dorothy that the Wizard can help them | them too” ] 
too [ All are convinced by Dorothy that the Wizard can help 

them too ] 


Try the query both with and without quotation marks around 
A phrase on the page: the sentence: 

[ “Dorothy uses the Golden Cap to summon the Winged 
Dorothy uses the Golden Cap to summon the Winged Monkeys” ] 
Monkeys [ Dorothy uses the Golden Cap to summon the Winged 

Monkeys ] 


Try the query both with and without quotation marks around 
A different phrase on the page: the sentence: 

[ “revealing an old man who had journeyed to Oz from 
revealing an old man who had journeyed to Oz from Omaha | Omaha long ago in a hot air balloon” ] 
long ago in a hot air balloon [revealing an old man who had journeyed to Oz from 

Omaha long ago in a hot air balloon ] 


3. Compare the pages you find that match the sentence or phrase. Is most of their MC the same? If so, does one 
clearly come from a highly authoritative source that is known for original content creation (newspaper, magazine, 
medical foundation, etc.)? Does one source seem to reasonably be the original? Does one source appear to 
have the earliest publication date, verified by sources like the Wayback Machine? 


Use your best judgment. Sometimes it is clear that the content is copied from somewhere, but you cannot tell what the 
original source is. Other times the content found on the original source has changed enough that searches for sentences 
or phrases may no longer match the original source. For example, Wikipedia articles can change dramatically over time. 
Text copied from old copies may not match the current content. If you strongly suspect the page you are evaluating is not 
the original source, consider it likely to be copied. 
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We have confirmed that the above text (the text in the sentence and phrases in the table) comes from an older version of 
a Wikipedia article about “The Wizard of Oz,” which can be found using the Internet Archive Wayback Machine. Please 
note that searching for the above sentence and phrases will not lead you to a current Wikipedia article about “The Wizard 
of Oz,” which demonstrates how confirming that the MC of a page contains copied content can sometimes be difficult. 


Important: The Lowest rating is appropriate if all or almost all of the MC on the page is copied with little or no time, effort, 
expertise, manual curation, or added value for users. Such pages should be rated Lowest, even if the page assigns 
credit for the content to another source. 


75 No Website Information 


As discussed in Section 2.5.3, we expect most websites to have some information about who is responsible for the 
website, as well as some contact information. For websites with YMYL pages, such as online banks, we expect to find a 
lot of information about the site, including extensive customer service information. 


Think about the purpose of the website and the type of website information users would expect or demand. 


YMYL pages with absolutely no website information, or other pages where the available information is completely 
inadequate for the purpose of the website (e.g., an online bank with only an email address), should be rated Lowest. 


7.6 Highly Untrustworthy, Unreliable, Unauthoritative, Inaccurate, or Misleading 


Sometimes, pages just don’t “feel” trustworthy. Use the Lowest rating for any of the following: 


e Pages or websites that you strongly suspect are scams (see these links for more information about Internet 
scams: Avoid Scams 1, Avoid Scams 2, Internet Fraud). 

e Pages that ask for personal information without a legitimate reason (for example, pages that ask for name, 
birthdate, address, bank account, government ID number, etc.). 

e Websites that “phish” for passwords to Facebook, Gmail, or other popular online services. See here for 
information about “phishing” fraud. 

e Pages with suspicious download links, which may be malware. 


If you suspect a link is malicious, please do not click on it. You do not need to personally experience a malicious 
download in order to confirm that a website is harmful. Likewise, if you feel strongly a website is a scam, you do not need 
to engage with it to get proof. 


Pages that appear highly untrustworthy should be rated Lowest, even if you’re not able to completely confirm their lack of 
trustworthiness. Please exercise caution and practice good Internet safety skills. 


7.7 Hacked, Defaced, or Spammed Pages on a Website 


Some websites are not maintained or cared for at all by their webmaster. These “abandoned” websites, especially 
websites that have become hacked, defaced, or spammed with a large amount of distracting and unhelpful content, 
should be rated Lowest. 


A hacked or defaced website is a site that has been modified without permission from the website owner(s). Responsible 
webmasters should regularly check their websites for suspicious behavior and take steps to protect users. 


We'll consider a comment or forum discussion to be “spammed” if someone posts unrelated comments that are not 
intended to help other users, but rather to advertise a product or create a link to a website. Frequently these comments 
are posted by a “bot” rather than a real person. Spammed comments are easy to recognize and may include Ads, 
download, or other links. Webmasters should find and remove this content because it is a bad user experience. 


While a specific page on a website may have a large amount of spammed forum discussions or spammed user 
comments, it does not mean that the entire website contains only spam. 
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7.8 Extremely Negative or Malicious Reputation 


Use the Lowest rating for websites with extremely negative reputations. Here are some examples originally shown in 
Section 2.7.4. All pages on these websites should be rated Lowest because of the extremely negative or malicious 
reputation of these websites. 


Website [Description S 


Extremely negative reputation information: This business has a BBB rating of F. There is a 
Site selling children’s jungle gym | news article about financial fraud. There are many reviews on websites describing how users 
sent money and did not receive anything in return. 


Site selling products related to Extremely negative/malicious reputation information: This website engaged in criminal 
eyewear behavior such as physically threatening users. 


Extremely negative reputation information: There are many detailed negative articles on news 
sites and charity watchdog sites describing fraud and financial mishandling about this 
organization. 


Organization serving the 
hospitalized veteran community 


7.9 Promotion of Hate or Violence 


Use the Lowest rating for pages created with the sole purpose of promoting hate or violence against a group of people 
based on criteria including (but not limited to) race or ethnicity, religion, gender, nationality or citizenship, disability, age, 
sexual orientation, or veteran status. Websites advocating hate or violence can cause real world harm. 


Reputation research is important for identifying websites that promote hate and violence. The Pew Research Center, the 
Anti-Defamation League, and the Southern Poverty Law Center are some reputable sources that can be used for 
reputation research. 


Some websites may not have reputation information available. In this case, please use your judgment based on the MC 
of the page and knowledge of your locale. 
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7.10 Examples of Lowest Quality Pages 


Type of Webpage/Content Lowest Quality Characteristics PQ Rating and Explanation 


Lowest Medium Highest 


This page is deceptively designed to make 
users believe they are on the official ABC News 
website. The official ABC logo is copied and 
used prominently throughout the page, and the 
URL of this website (http://abcnews.com.co/) is 
very similar to the official site's URL 
(http://abcnews.go.com/). 


Harmful purpose 


Lowest: Deceptive page purpose Deceptive design: impersonates a 


and design — News website different website 
(YMYL) Lowest quality MC: demonstrably 


inaccurate content The articles on this website include 


demonstrably inaccurate content, such as an 
inaccurate news report that Obama banned the 


Pledge of Allegiance. 


The deceptive website design and inaccurate 
content can misinform users. This website was 
created with a harmful purpose. 


Lowest Medium Highest 


The MC on this page contains factually 
unsupported theories related to the Vatican's 
knowledge of the planet Nibiru, the existence of 


Presents unsubstantiated conspiracy aliens, and upcoming world events. Although 


Lowest: Deceptive page purpose theories as if the information were various Vatican officials and scientists are 
— Unsubstantiated conspiracy factual quoted in the article, the quotations do not 
theories Lowest quality MC: demonstrably support the article's claims, and in some cases 


Inaccurate content do not seem to come from the person quoted. 


The Nibiru cataclysm and related events have 
also been thoroughly debunked by authoritative 
sources (Reference 1, Reference 2). The 
demonstrably inaccurate content on this page 
can misinform users. 


Lowest Medium Highest 


At the time this example was written, Betty 
White was alive. This profile page on a social 
media website leads users to believe she is 


; F e Harmful purpose dead. Comments on the page indicate that 
Lowest: Demonstrably inaccurate À . ; E 
e Lowest quality MC: demonstrably some users have realized that it is inaccurate. 


= : : : l ! 
content — Social media profile inaccurate content If the page were truly created in error, these 


comments should have led the creator to 
realize the mistake and delete the page. 


While it is not clear why this type of page was 
created in the first place, it appears to be a 
deliberate intent to misinform users, which is a 
harmful purpose. 
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Type of Webpage/Content 


Lowest: Demonstrably inaccurate 
content — News article 


Lowest: Keyword Stuffing -Forum 
page: Watch RoboCop online 


Lowest: Copied content — Wizard 
of Oz 


Lowest: Unsatisfying amount of 
MC — Wedding dresses 


Lowest: Malicious, harmful, or 


deceptive — Asks for personal 
information (YMYL) 


Lowest Quality Characteristics PQ Rating and Explanation 


Harmful purpose 

Deceptive page design 

Lowest quality MC: demonstrably 
inaccurate content 

An unsatisfying amount of information 
about the website and author 


e Lowest quality MC: large amount of 
“Keyword Stuffed” MC 


Lowest quality MC: copied content with 
little or no time, effort, expertise, manual 
curation, or added value for users 

No evidence of E-A-T 


An unsatisfying amount of MC for the 
topic of the page 

Very low quality MC 

No evidence of E-A-T 

True purpose of the page and website is 
questionable 


Malicious, harmful, or deceptive 
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Lowest Medium Highest 


This news article is demonstrably inaccurate. 
The event described here (the death of Miley 
Cyrus) did not happen. Note that no date is 
given, no sources are cited, and there is no 
author. This website is designed to look like a 
news source, but there is no information on the 
news organization that created the website or 
its content. The SC also features distracting 
pictures and outrageously titled links. 


This page was created to misinform users, 
which is a harmful purpose. 


Lowest Medium Highest 


This is an example of a spammed forum page, 
where the goal is to try and get users to watch 
a movie online, which has nothing to do with 
the purpose of the site. For a forum page, the 
user comments are the MC, and much of the 
content is not helpful for users. Here is an 
example of some of the “keyword stuffing” on 
this page that is not meant to be read by a 
human, found near the bottom of the page. 


Lowest Medium Highest 


All MC is a copy of an old Wikipedia article, 
which you can see here by researching it on 


the Internet Archive Wayback Machine. 


The current Wikipedia page offers much more 
content, links, references, etc. This page 
doesn’t add much value for users. 


— | | | 


Lowest Medium Highest 


This page has so many low quality 
characteristics that the Lowest+ rating is 
appropriate. There is a question about what 
the true purpose of the page is—Lowest is 
appropriate if you feel this page was created 
with little attempt to help users. 


Lowest Medium Highest 


This is a shopping checkout page, but it asks 
for the user’s government ID number, driver’s 
license number, ATM pin number, bank account 
information, etc. 
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Type of Webpage/Content Lowest Quality Characteristics PQ Rating and Explanation 


Lowest Medium Highest 


The first sentence is incomplete, and doesn’t 
provide an explanation for the code, and 
therefore fails the purpose of the page. Other 
pages on the website have inaccurate, 
misleading, and/or incomplete information. In 
fact, some pages seem borderline gibberish, 
with contradicting information. It appears the 
page exists only to make money, with no 
attempt to help users. 


Auto-generated MC, created with little to 
no time, effort, expertise, manual 
curation, or added value for users 

No evidence of E-A-T 


Lowest: Auto-generated with Little 


Attempt to Help Users — Engine 
Code (YMYL) 


Lowest Medium Highest 


Lowest: Fraudulent — Charity e Extremely negative or malicious 


(YMYL) reputation There are many claims of fraud and financial 


problems about this organization on reputable 
news sites and charity watchdog sites: 


Negative review 1, Negative review 2, Negative 
review 3, and Negative review 4. 


Lowest Medium Highest 


There is no information about who created this 
website, no contact information, and no 


Lowest: No website information — No website information for YMYL website eae . 
E Sa. RA authorship information. 


About dry sockets (YMYL) No evidence of E-A-T 


Medical pages require a high degree of user 
trust. Because there is no information about 
who owns this website and who created this 
content, we will consider this an untrustworthy 
website. 


Lowest Medium Highest 


Deceptive purpose (fake directory page) 

No website information for YMYL topic The links on the page appear to link to helpful 

No evidence of E-A-T articles, but in fact the links are Ads that do not 
go to articles. This is a YMYL topic, but there is 
no indication about who is responsible for this 
content or website. 


Lowest: Fake directory page - 
Mesothelioma (YMYL) 


Lowest Medium Highest 


Lowest: Fake search page — 


Chicken recibes Deceptive purpose (fake search page) 


Advertising should never disguise itself as the 
MC of the page. Pages with Ads that are 
designed to look like MC should be considered 
deceptive. 


Lowest Medium Highest 
Lowest: Fake search page — ; 


Chairs Deceptive purpose (fake search page) 


This page was created to make money from 
clicks on Ads rather than to help users. 
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Type of Webpage/Content Lowest Quality Characteristics PQ Rating and Explanation 


Lowest Medium Highest 


This looks like an information website for a 
drug. The real purpose of this page is to get 
users to this website, and then get them to click 
on the “order now” link, which takes users to 
another website. 


Lowest quality MC (copied content with 
little or no time, effort, expertise, manual 
curation, or added value for users) 

No website information for YMYL website 
No evidence of E-A-T 


Lowest: Deceptive purpose — 
Viagra (YMYL) 


All MC on this page is copied with little or no 
time, effort, expertise, manual curation, or 
added value for users. Here is the original 
source. 


Lowest Medium Highest 


Purpose of the page to make money with 
Lowest: Deceptive page purpose very little or no attempt to help users This page is titled “Washing Machine Reviews,” 
and design — Washing machine Lowest quality MC (copied content with but there are no reviews on the page and the 
reviews little or no time, effort, expertise, manual content is copied from another website. The 
curation, or added value for users) links all go to a single Internet retailer selling 
washing machines. This is a deceptive page 
because it is designed to get users to click on 
the prominent links. 


Lowest Medium Highest 


This page contains a fake survey, which 
appears to promise users the opportunity to win 
an Apple MacBook Air, an iPhone 5c, ora 
Galaxy S4. However, after completing some 
survey questions and selecting the product the 
user would like to win, the user is redirected to 
website after website to complete more and 
more survey questions, while being asked to 
supply more and more personal information, 
including medical information and bank account 
information. 


Lowest: Deceptive page purpose e Deceptive purpose 
— Fake survey page e Highly untrustworthy 


There is no information about who is 
responsible, even though this website asks for 
highly personal information such as bank 
account numbers. Despite appearances, there 
is no affiliation with Google. 


Lowest Medium Highest 
Lowest: Deceptive page design — 


Javascript alert box e Deceptive or misleading page design 


Some users might not even notice the MC 
because it is under a long list of Ads. Users 
may mistake the Ads for MC. 
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Type of Webpage/Content Lowest Quality Characteristics PQ Rating and Explanation 


Lowest Medium Highest 


The title of this page is “Rachael Ray Diet 
Blog,” but the page has nothing to do with 
Rachael Ray or her diet or her products. This 
page exists to sell products using Rachael 
Ray’s name and image. In fact, there is a 

e Deceptive purpose and deceptive content | brown-text-on-brown-background section at the 
bottom of the page (which we consider to be 
hidden text) that says “Disclaimer: Rachael Ray 
is not affiliated with nor does she sponsor or 
endorse this blog.” This page is deceptive in 
spite of the disclaimer! 


Lowest: Deceptive page purpose 
— Rachael Ray diet blog 


This example has been annotated with red text 
at the top to point out deceptive aspects of this 
page. 


| 
Lacking in purpose Medium Highest 
Lowest quality MC (gibberish, keyword 
stuffing) 


Lowest: Keyword stuffing and 
gibberish- Imodium for dogs 
This page has no helpful MC and no helpful 
purpose. 


Lowest Medium Highest 


Lowest: Malicious website — Site 


selling products related to 
eyewear (YMYL) 


Very negative, malicious, This website took users’ money and physically 

or financially fraudulent reputation threatened users who complained. These 
articles on Wikipedia and the New York Times 
describe the deceptive techniques used by this 
website and provide other negative information 
about the website and its owner. 


vw Medi Highest 
Lowest: Gibberish — Celebrity Lacking in purpose — a — 


site Lowest quality MC (gibberish) 
This page has no helpful MC and no helpful 
purpose. 


Lowest Medium Highest 


Lowest: Gibberish — PDF file Lowest quality MC (gibberish) 


Lack of purpose This is a gibberish PDF file. We don’t have any 
idea why this was created. There are no links 
or Ads. This page has no purpose and no 
reason to exist. 


| 
Lowest Medium Highest 
Lowest: Lack of purpose — Free Lowest quality MC (gibberish) 


eee er) Highly untrustworthy This page has deceptive features, such as a 


friend request, a prize alert, and a download 
button. 
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Type of Webpage/Content Lowest Quality Characteristics PQ Rating and Explanation 


Lacking in purpose Lowest Medium Highest 


Lowest quality MC (gibberish, keyword 
stuffing) 


Lowest: Lack of purpose — Las 
Vegas gambling terms 


This page has no MC and no helpful purpose. 


Lowest Medium Highest 
Lowest: Highly untrustworthy — No evidence of E-A-T 
Article about how to cure Inaccurate or misleading 


stomach flu (YMYL) No website information for YMYL topic ls Content 1S Bea ye ay ang USSSA lot or 


words to say very little. There is no information 
about who is responsible for the content and no 
contact information for this YMYL medical topic. 


Lowest Medium Highest 


This is a download website with only one page 
— this page. There is a lot of text (with grammar 
and spelling errors) promising that users will 
make money from this free download. The 
purpose of the page seems to be to entice 
users into clicking on the links with the promise 
of making money. 


Lowest: Highly untrustworthy — 


Download pade Highly untrustworthy (suspect download) 


Remember that you are not required to click on 
any links that are suspicious. 


Lowest Medium Highest 


This is an example of a page with no MC. You 
might think that the MC is “missing” due to a 
problem with this particular page, but in fact, 
this website has hundreds of pages that look 
the same way—no MC, just Ads. This website 
shows Ads with little or no attempt to help 
users, and should be rated Lowest quality. 


Lowest: Deliberately created with Large amount of prominent Ads and no 
no MC attempt to help users 


— | | | 


Lowest Medium Highest 


Lowest: Low quality MC — Article | e Lowest quality MC This content has many problems: poor 


grammar, and sentences that are meaningless 
or state something obvious. For example: 
"Popping pimples could be or could be not the 
new trend of getting rid of them." In addition, 
the “About the Author” section of the page has 
an Ad, but no information about the author. 


about popping pimples e No evidence of E-A-T 


— | | | 


Lowest Medium Highest 


Lowest: Low quality MC — Article : 
i 7 e Lowest quality MC : y 
about getting a mortgage in Texas ; A- The level of expertise of the author of this 
e No evidence of E-A-T i - P 

(YMYL) content is not clearly communicated. Providing 
this background information is particularly 
important for medical, financial, or other YMYL 
topics for which expertise is needed. 
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Type of Webpage/Content Lowest Quality Characteristics PQ Rating and Explanation 


=æ | | | 


Lowest Medium Highest 


Lowest: YMYL page with 

inaccurate information — Chest YMYL page with inaccurate potentially We must evaluate this page from the point of 

pains and smoking (YMYL) dangerous medical advice view of a user visiting this page from a search 

Lowest quality MC engine, rather than a participant. The question 

is poorly worded and difficult to understand. 
The answers are poorly worded and have 
incorrect and potentially dangerous medical 
advice. The MC is low quality. 


Lowest Medium Highest 


YMYL page with potentially damaging 


Lowest: YMYL page with bad ; : y 

information — Paying off loans financial advice i i i 

anne e No evidence of E-A-T This page gives loan advice that can be l 

( ) Inaccurate or misleading potentially damaging, for example, instructing 
people not to pay back their loans. The article 
has grammar and spelling errors, and the page 
is highly untrustworthy. 


æ | | | 


Lowest Medium Highest 


This page is selling Nike Air Jordan shoes. 
When you look at the “Contact Us” page, it 
does not give the name of a company or a 
Completely inadequate or untrustworthy | physical address, which also cannot be found 
Lowest: Potentially untrustworthy customer service information for a anywhere else on the website. This amount of 
shopping page with insufficient shopping website contact information is not sufficient for a 
contact info (YMYL) No evidence of E-A-T shopping website. 
Inaccurate or misleading 
In addition, the “Shipping and Returns” page 
has the name of another company that seems 
to be unrelated. There are also official looking 
logos at the bottom of the homepage, including 
the Better Business Bureau logo and Google 
Checkout logo, that don’t appear to be 
affiliated with the website. 


8.0 Medium Quality Pages 


In this section, we will describe pages that should get the Medium quality rating. Medium pages achieve their purpose 
and have neither high nor low expertise, authoritativeness, and trustworthiness. However, Medium pages lack the 
characteristics that would support a higher quality rating. Occasionally, you will find a page with a mix of high and low 
quality characteristics. In those cases, the best page quality rating may be Medium. 


The page achieves its purpose, however, it does not merit a High quality rating, nor is there 
anything to indicate that a Low quality rating is appropriate. 


The page or website has some characteristics of both High and Low quality pages, but the low 
Mixed, but has some E-A-T quality characteristics are mild enough that the convincing high quality aspects make it difficult to 
rate the page Low. 
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8.1 Examples of Medium Quality Pages 


Webpage’/Type of Content 


Medium: Encyclopedia 


Wikipedia article about baroque 
pearls 


Medium: Humor 


Page from a humorous site 


Medium: Entertainment 1 


Article about “Keeping Up with the 
Kardashians” show 


Medium: Entertainment 2 


Article about Miley Cyrus 


Medium: Q&A 


Q&A page where a user is looking 
for advice on where to buy high 
quality women's clothing online 


Nothing wrong, but nothing special 


Nothing wrong, but nothing special 


Nothing wrong, but nothing special 


Nothing wrong, but nothing special 


Nothing wrong, but nothing special 
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Medium Quality Characteristics PQ Rating and Explanation 


Medium Highest 


This is a short Wikipedia article about baroque 
pearls, a fairly narrow topic. This page is OK 
for its purpose, but it doesn’t display 
characteristics associated with a High rating. 


Lowest Medium Highest 


This page is from a humorous site that 
encourages users to post photos with mouths 
drawn on them. This page is OK for its 
purpose, but it doesn’t display characteristics 
associated with a High rating. 


Medium Highest 


This page is on a website dedicated to 
entertainment news. This page is OK for its 
purpose, but it doesn’t display characteristics 
associated with a High rating. 


Lowest Medium Highest 


This page is from a news/entertainment 
website. This page is OK for its purpose, but it 
doesn’t display characteristics associated with 
a High rating. 


Lowest Medium Highest 


This Q&A page has a discussion of different 
online merchants. There is some everyday 
expertise, but it doesn’t display characteristics 
associated with a High rating. 
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Webpage’/Type of Content 


Medium: Page with Error 
Message 


“Custom 404” page 


Medium: Informational 
Page about propulsion on the 


“Quality Reasoning Group” 
section of a university’s website 


Medium: Lyrics 


Song lyrics for the song “Never 
You/Fear Love” 


Medium: Recipe 1 


Mexi-Chicken Casserole on a 
newspaper website 


Medium: Recipe 2 


Recipe for cherry-topped cake 


e Nothing wrong, but nothing special 


Mixed, but has some E-A-T 

Website has a good reputation 
Content is likely to be accurate and 
trustworthy because of the website, 
though no reference links are provided 


Nothing wrong, but nothing special 


e Nothing wrong, but nothing special 


e Nothing wrong, but nothing special 
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Medium Quality Characteristics PQ Rating and Explanation 


Lowest Medium Highest 


This is an example of a “custom 404” page. 
These pages are designed to alert users that 
the URL they are trying to visit no longer exists. 
Some websites do a nice job of not only 
alerting users about a problem, but also giving 
them help. 


This page is on a well-known merchant website 
with a good reputation. However, this particular 
page displays the bare minimum of content 
needed to explain the problem to users, and 
the only help offered is a link to the homepage. 


Lowest Medium Highest 


Although this is a well-known, highly-respected 
university with a high quality site, this page is 
on a very specialized section of the university 
website. No author is listed and the page may 
have been a one-time project, possibly from a 
student, which is no longer maintained. 


Lowest Medium Highest 


There are many lyrics websites that have 
similar content. This page is OK for its 
purpose, but it doesn’t display characteristics 
associated with a High rating. 


Lowest Medium Highest 


This recipe was contributed by an author of 
cookbooks. However, the page has no SC 
related to the purpose of the page, such as 
reviews or links to other recipes, etc. This page 
is OK for its purpose, but it doesn’t display 
characteristics associated with a High rating. 


Lowest Medium Highest 


This website is known for high quality content 
about animals and the environment. This 
particular page has a recipe for kids. There 
isn’t much MC or SC. This page is OK for its 
purpose, but it doesn’t display characteristics 
associated with a High rating. 
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Webpage’/Type of Content Medium Quality Characteristics PQ Rating and Explanation 


| | 
Lowest Medium Highest 
Medium: Video 


e Nothing wrong, but nothing special oe : , 
Video of a kitten meowing a lot This is a professionally-created video of a cute 


kitten meowing. This page is OK for its 
purpose, but it doesn’t display characteristics 
associated with a High rating. 


Medium: Forum 1 oe Medium Highest 


Forum page on an online auction e Mixed, but with some redeeming qualities 

website. A user is looking for The relative lack of MC is balanced a bit by the 

help choosing a product category. expertise of this forum. This forum is dedicated 
to this kind of question. 


Lowest Medium Highest 


This forum is about dance topics, and many 
pages have expertise from a community of 
Medium: Forum 2 ballet dancers. On this particular page, 
aa aa participants have everyday experience washing 
Forum page on a dance website. | e Mixed, but with some redeeming qualities | ballet shoes and make recommendations 
A user is looking for advice on based on their own experiences. 
how to wash ballet shoes. 
This page is “mixed” because there is some 
distracting content that makes it hard to read 
the MC. However, this is not a beauty contest! 
Even though the page may be cluttered, there 
is some valuable everyday expertise and 
helpful MC, making Medium a good rating for 
this page. 


9.0 Page Quality Rating Tasks 


At first, PQ rating may seem difficult. There are several aspects of the page and the website to look at and think about. 
This type of rating takes practice. Rereading sections of these guidelines and thinking about the examples may help 
when you encounter difficult rating tasks. 


Important: Do not struggle with each PQ rating. Please give your best rating and move on. If you are having trouble 
deciding between two ratings, use the lower rating. If you are torn between three ratings, choose the one in the middle. 


Do not consider the country or location of the page or website for PQ rating. For example, English (US) raters should use 
the same PQ standards when rating pages from other English language websites (UK websites, Canadian websites, etc.) 
as they use when rating pages from U.S. websites. In other words, English (US) raters should not lower the PQ rating 
because the page location (UK, Canada) does not match the task location. 


These guidelines are specific to “regular” webpages. Occasionally, you may be asked to rate a landing page that is nota 
webpage. For example, you may be asked to rate a PDF file, a PNG or JPEG image file, etc. When the landing page of 
the URL is not a webpage, some of the criteria in these guidelines may not apply. In this case, please use your judgment. 


Finally, this Page Quality Rating Guideline does not completely cover every aspect of page quality. If you find pages that 


you truly believe to be High or Low quality, please rate them as such, even if the reason is based on something not 
covered in this document. Please use the comment section to explain your reasoning. As always, use your judgment. 
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9.1 Instructions for Rating Page Quality Tasks 
The Page Quality task page is broken up into several parts: 


1. Some initial questions about the task landing page. 

2. A “PQ grid” to record your observations about PQ characteristics of the landing page. 
3. The Overall PQ rating slider which records your Overall PQ rating. 

4. Acomment box to explain your rating. 


Some results to the initial questions will end the task early. If the page is Porn, Foreign Language, or Didn’t Load, you will 
not fill in the PQ grid or assign an overall rating. 


e Foreign Language should be used when the language on the landing page is NOT the task language, a language 
that is commonly used by a significant percentage of the population in the task location, or English. 

e Didn't Load should be used for pages where there is absolutely no content on the page created by the website. 
There is no MC, SC, or Ads on the page. See this Wikipedia article for descriptions of different types of error 
messages. 


Similarly, if you respond that the page is malicious, harmful, deceptive, or lacking in purpose, you will also not fill in the PQ 
grid or assign an overall rating. 


The PQ grid is designed to be your "note pad." It allows you to record your observations about the landing page and the 
website it belongs to. 


9.1.1 Rating on Your Phone 


You should open the task landing page on your phone using the standard Send to Device feature, unless otherwise 
instructed. However, more intensive analysis on the website can be done on your computer. For example, you should 
open and explore the links that are provided to help you do research on website information and reputation on your 
computer. 


Important: Clicking on the task URL may bring up an interstitial page. You can ignore this page in your rating criteria if 
you can easily get to the MC. However, if the interstitial page makes it extremely hard (or impossible) to get to the MC, 
that should factor into your Page Quality rating. 


9.2 E-A-T: Page or Website? 


The quality of the MC is evaluated by looking at the landing page of the link in the PQ rating task. The reputation of the 
website is based on the website that the landing page belongs to. Depending on the page, E-A-T may be based on the 
page alone, may be based on the website, or may be based on both the page and website. 


Landing page E-A-T is important when a website has different authors on different pages. This is the case for article 
websites or websites like YouTube, which have user-generated content. E-A-T for pages on these websites may differ 
drastically based on the E-A-T of the creator of the content on the page. 


Website E-A-T is important in the following situations: 


e All content on the website is produced by the same person or organization. An example is a medical website that 
is produced by a reputable physician group. 

e The content of the website is produced by different authors or organizations, but the website has very active 
editorial standards. An example of this is a science journal with very high standards for publication. 

e The website has an extremely positive reputation from experts in the topic of the website, i.e., the website is 
acknowledged to be one of the most expert, authoritative, or trustworthy sources on the topic. 
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10.0 Page Quality Criteria for Specific Types of Pages 
10.1 Ratings for Encyclopedia Pages 


There are many encyclopedia-type websites. Some are highly-respected publications that are standard references, while 
some are websites with content created and edited by anonymous users, with no editorial oversight or fact checking. We 
may not always know the author of the specific encyclopedia article, and therefore must rely on website reputation 
research to determine the E-A-T of the article. High and Highest quality ratings should only be used for encyclopedias 
with very good reputations for accuracy and expertise, where the article itself is well-researched with appropriate 
references cited. 


A note about Wikipedia: in general, the website has a good reputation and is a very popular resource that is generally 
valued for accuracy. However, there is no single author or organization that vouches for the accuracy of Wikipedia 
articles, and the quality of pages varies. You should perform page-level checks on individual articles. 


A Wikipedia article on a non-YMYL topic (example) with a satisfying amount of accurate information and trustworthy 
external references can usually be rated in the High range. Some Wikipedia articles may even be rated higher, although 
remember that a high level of expertise is required for YMYL topics. A PQ rating in the Medium range is often appropriate 
for pages with less extensive MC and external references. Naturally, Wikipedia articles with very little MC should get 
lower PQ ratings. Factual inaccuracy is a sufficient reason for a Low or Lowest rating. 


10.2 Ratings for Pages with Error Messages or No MC 


Some pages are temporarily broken pages on otherwise functioning websites, while some pages have an explicit error (or 
custom 404) message. In some cases, pages are missing MC as well. Please think about whether the page offers help 
for users—did the webmaster spend time, effort, and care on the page? 


Here are some examples. 


Webpage’/Type of Content Characteristics of the Page PQ Rating and Explanation 


Lowest Medium Highest 


Large amount of prominent Ads and no This is an example of a page with no MC. You 


Deliberately created with no MC might think that the MC is “missing” due to a 
attempt to help users i f $ ; 
problem with this particular page, but in fact, 


this website has hundreds of pages that look 
the same way—no MC, just Ads. This website 
shows Ads with little or no attempt to help 
users, and should be rated Lowest quality. 


Lowest Medium Highest 


No MC (probably due to a temporary 
Page didn't fully load and has no technical error) This page has no MC and no error message. It 
MC Sufficient SC is an isolated example of a page with no MC or 
Positive reputation error message on a website for a reputable 
newspaper for a town in Michigan. All of the 
navigation links work, and the page was later 
fixed. 
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Webpage’/Type of Content Characteristics of the Page PQ Rating and Explanation 


Lowest Medium Highest 


This is an example of a “custom 404” page, 
alerting users that the URL they are trying to 
visit no longer exists. Some websites do a nice 
job of alerting users about a problem and 
providing helpful tips. 


Error page with custom 404 e Nothing wrong, but nothing special 

message 1 
This page is on a well-known merchant website 
with a good reputation. However, this particular 
page displays the bare minimum of content 
needed to explain the problem to users, and 
the only helpful content is a link to the 
homepage. 


| | pj 


A satisfying amount of high quality MC for | “= Minia Highest 


its purpose 

Helpful SC that improves the user This is an example of a “custom 404” page, 

experience alerting users that the URL they are trying to 

Positive reputation visit no longer exists. This website does a nice 
job of explaining the issue and providing helpful 
tips, including a search box. 


| | p 


Lowest Medium Highest 


Error page with custom 404 
message 2 


This is an example of a “custom 404” page. 
These pages are designed to alert users that 
the URL they are trying to visit no longer exists. 
The MC of this page is the cartoon, the caption, 
and the search functionality, which is specific to 
the content of the website. Itis clear that time, 
effort, and talent was involved in the creation of 
the MC. 


A satisfying amount of high quality MC for 
its purpose 

Helpful SC that improves the user 
experience 

Very positive reputation 


Error page with custom 404 
message 3 


This publication has a very positive reputation 
and is specifically known for its cartoons, which 
allows us to go as high as High+ to Highest. 


10.3 Ratings for Forums and Q&A pages 


Ratings for forum and Q&A pages can be challenging. The most important aspect is the E-A-T of the participants in the 
discussion, which can be difficult to judge. Keep in mind the following: 


e The Main Content on forum and Q&A pages includes both the question as well as the answers/responses and 
resulting discussions. 

e Rate forum and Q&A pages from the point of view of a user who visits the page, rather than a participant involved 
in the discussion. 
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Here are some examples. 


Webpage/Type of Content Characteristics of the Page PQ Rating and Explanation 


Q&A page about chest pains and 


smoking (YMYL) e YMYL page with inaccurate potentially 


dangerous medical advice 


e Misleading page design 
e Unsatisfying amount of MC for the 
purpose of the page 


Q&A page with unanswered 
question 


e Misleading page design 
e  Unsatisfying amount of MC for the 
purpose of the page 


Q&A page about a 2002 Volvo 
part 


Q&A page about a game console e  Unsatisfying amount of MC for the 
issue purpose of the page 
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=æ | | | 


Lowest Medium Highest 


We must evaluate this page from the point of 
view of a user visiting this page from a search 
engine, rather than a participant. The question 
is poorly worded and difficult to understand. 
The answers are poorly worded and have 
incorrect and potentially dangerous medical 
advice, making it lowest quality MC. 


Lowest Medium Highest 


In addition to having no answer, this page has 
Ads and links to other questions (misleadingly 
labeled as “Relevant answers”) displayed 
prominently, which users may mistake for 
answers to the question. It takes a moment to 
notice that this page actually has no answer. 
Deceptive design and lack of an answer make 
this page a frustratingly poor user experience 
and cause this page to completely fail to 
achieve its purpose. 


| =| | | 
Lowest Medium Highest 


In this example, the MC is boxed in red. Please 
read the MC, including the completely unhelpful 
"answer" to the question in the red box. This 
answer is so unhelpful, we can consider this 
question to be unanswered. This page has an 
unsatisfying amount of MC. 


In addition to a very unhelpful “answer,” the 
page design makes it difficult to distinguish the 
MC from Ads. For example, below the answer, 
we see a "sponsored answer," which has the 
same format as the real answer, but is actually 
an Ad and not an answer to the question. This 
page design is somewhat misleading. 


Lowest Medium Highest 


Some websites rely on users to create virtually 
all of their MC. In this case, the MC is the 
user’s question. If there are no answers, the 
amount of MC on the page is unsatisfying. 
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Webpage’/Type of Content Characteristics of the Page PQ Rating and Explanation 


Lowest Medium Highest 


Q&A page about Native American | e Lacking E-A-T for the purpose of the 

customs page There are 94 answers to this question with a 
few results that seem helpful. Many of the 
posts are wrong or misleading, including the 
top answer, which is labeled the “best answer.” 


Lowest Medium Highest 


This forum is about dance topics, and many 
pages have expertise from a community of 
ballet dancers. On this particular page, 

Forum page on how to wash participants have everyday experience washing 


ballet shoes e Mixed, but with some redeeming qualities | ballet shoes and make recommendations 
based on their own experiences. 


This page is “mixed” because there is some 
distracting content that makes it hard to read 
the MC. However, this is not a beauty contest! 
Even though the page may be cluttered, there 
is some valuable everyday expertise and 
helpful MC, making Medium a good rating for 
this page. 


Lowest Medium Highest 


Q&A page about whether a e Everyday expertise for the purpose of the | Many participants share their personal 

Roomba will work page experiences with these products, giving details 
such as how well certain models work with pet 
hair. There are many descriptions of 
participants’ own experiences with this product 
and how well it works for them. 


Lowest Medium Highest 


The answer on this page is written by an 
Q&A page about interviewing at e Everyday expertise for the purpose of the | employee of Build-A-Bear, so it offers a 
Build-A-Bear page somewhat unique and presumably somewhat 
expert answer. 


The page design allows users to read the MC 
as the Ads are clearly labeled, as well as both 
the question and answer. 


Lowest Medium Highest 


The person who posted the first message on 
this forum page provides a helpful resource on 
how to master the magic loop technique in 
knitting. She also shares pictures of her own 
version using an old pair of blue jeans. With 
over 20 years of experience knitting socks, we 
would consider her to be an expert on the topic. 


Forum post on the magic loop High level of E-A-T for the topic and 
technique in knitting purpose of the page 
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Webpage’/Type of Content 


Everyday expertise 
A satisfying amount of high quality MC 


Q&A page about how long most e 
cancer patients live e 


e High level of E-A-T for the topic and 
purpose of the page 
e A satisfying amount of high quality MC 


Forum page on authenticating a 
purse 


e High level of E-A-T for the topic and 
purpose of the page 
e A satisfying amount of high quality MC 


Forum page on KPIs to track 


High level of E-A-T for the topic and 
purpose of the page 
A satisfying amount of high quality MC 


Forum page on landscaping an 
aquarium 


Very high level of E-A-T for the purpose 
of the page 

Very positive reputation, and the website 
is responsible for all content on the 
website 


Q&A page on abbreviations 


Copyright 2017 


Characteristics of the Page PQ Rating and Explanation 


Lowest Medium 


— 


Highest 


The question on the page asks how long 
people live with cancer. There are many 
results describing how long a loved one lived 
after diagnosis. There is very little medical 
advice and the focus of the page is sharing 
personal experience. Many results are heartfelt 
and well written. 


Lowest Medium 


p 


Highest 


This forum is well-known for discussions on 
luxury designer purses. On this particular 
forum page, members are consulting forum 
experts who have expertise authenticating bags 
from this brand. These experts can tell if a 
particular bag is authentic or fake. While there 
is an ad at the top and a few ads within the 
forum message, it does not distract from the 
MC, which is easy to find. 


Lowest Medium 


— | 


Highest 


This forum is well-known for its community of 
experts on quality assurance. The responses 
are authoritative and trustworthy for the 
purpose of the page. While there are Ads on 
the page, it is easy to find the MC and they are 
clearly labeled as “Sponsored Link.” 


Lowest Medium 


p 


Highest 


This discussion focuses on landscaping for a 
particular paludarium (an aquarium with 
terrestrial and aquatic elements). There is a lot 
of discussion and interaction between forum 
members about the types of materials and 
species used in the aquarium. The posts show 
expertise the niche topic of aquarium 
landscaping. 


Lowest Medium 


= 


Highest 


The purpose of the page is to answer questions 
about the usage of abbreviations. This website 
has a very good reputation as a reference for 
information on writing, publishing, etc. Itis 
considered highly authoritative and trustworthy 
for the topic of the page. 
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11.0 Page Quality Rating FAQs 


Why do we have to do all 
these steps? This takes a 
long time! 


Are we just giving High 
quality ratings to pages that 
“look” good? 


You talked about expertise 
when rating MC. Does 
expertise matter for all 
topics? Aren't there some 
topics for which there are 
no experts? 


Aren't there some types of 
pages or topics, such as 
celebrity gossip, that always 
have Low quality content? 


I've never seen a High 
quality page of type X. If 
there are no high quality 
pages of this type, why are 
we giving existing pages a 
Low quality rating? 


Some of these criteria seem 
unfair. For example, some 
art pages do not have a 
purpose. Are these pages 
Low quality? 


If | have to open the task 
URL on my phone, can | 
check for E-A-T on my 
desktop/laptop? 


With practice, the amount of time needed for accurate PQ ratings will decrease. The 
steps are important and are designed to help you assess many different aspects of PQ. 
You may be surprised by what you find. Pages that initially look Low quality may turn 
out to be Medium or High quality with careful inspection. The reverse may happen as 
well. We want your most informed, thoughtful opinion. 


No! The goal is to do the exact opposite. These steps are designed to help you 
analyze the page without using a superficial “does it look good?” approach. 


Remember that we are not just talking about formal expertise. High quality pages 
involve time, effort, expertise, and talent/skill. Sharing personal experience is a form of 
everyday expertise. 


Pretty much any topic has some form of expert, but E-A-T is especially important for 
YMYL pages. 


For most page purposes and topics, you can find experts even when the field itself is 
niche or non-mainstream. For example, there are expert alternative medicine websites 
with leading practitioners of acupuncture, herbal therapies, etc. There are also pages 
about alternative medicine written by people with no expertise or experience. E-A-T 
should distinguish between these two scenarios. 


For almost any type of page or informational topic, there is a range of content quality. 
Remember that high quality content is defined as content that takes time, effort, 
expertise, and talent/skill. Pages that have a harmful purpose should be rated Lowest 
quality, regardless of their topic. 


For example, there are both High and Low quality celebrity gossip pages. Often, the 
purpose of these pages is to share scandalous, but potentially true personal information 
about celebrities. We can consider the MC of a gossip page to be high quality if it is 
interesting information from a somewhat plausible source. 


For some topics or types of pages, there may not be many (or any!) High quality pages 
now, but there may be in the future. We need a uniform set of standards that apply to 
all pages, even for pages that have not yet been created. 


Art pages do have a purpose: artistic expression. Pages created for artistic expression 
do not deserve the Low quality rating simply because they have no other purpose. 
Artistic expression, humor, entertainment, etc. are all valid and valued page purposes. 


Yes! More intensive analysis on the website (e.g., researching E-A-T) can be done on 
your computer. Any time that using your computer will help you complete the task, feel 
free to use it. Also, tasks should provide some links to help you do website research, 
and there is a note directing you to do it on your desktop/laptop. 
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Part 2: Understanding Mobile User Needs 


12.0 Understanding Mobile Users, Mobile Queries, and Mobile Results 


What do you do on your mobile smartphone? 


Check the weather 
Send a text message Call Mom 


Place a bid 


Watch a video Read the news 
Check updates 


Set an alarm Play a game 
Check in for a flight 
Find a gas station Find an address 
Read a book 
Send a tweet Listen to a song 


Get directions 
to a coffee shop 


| want to... 


People rely on their phones for many different tasks in different environments. Users may want to search the web or may 
want to tell the phone to do something specific. Keep in mind that tasks can be simple or complex, and may take multiple 
steps to complete. For example, a simple task may be to find the director of a movie. A complex task may be to find a 
movie’s showtimes nearby, purchase tickets, get directions, and then use the phone’s navigation to go to the theater. 


We expect our phones to do a lot. At the same time, phones can be challenging to use, especially compared to a desktop 
computer or laptop: 


e Entering data may be cumbersome: typing is difficult on mobile smartphones, and when users speak to their 
phones instead of typing, voice recognition may not always be accurate. 

e Small screen sizes make it difficult to use some phone features, apps, and webpages. 

e Some webpages are difficult to use on a mobile phone. Website navigation can be difficult as menus and 
navigation links may be small. Webpages may require left-to-right scrolling to read text. Images may not fit on 
the screen. In addition, many mobile devices cannot access webpages with Flash or other similar features. 

e Internet connectivity can be slow and inconsistent for mobile users going in and out of networks. App 
opening, recognition of voice commands, and webpage load times can be very slow on a mobile phone. 


Important: Mobile smartphones should make tasks easy, even for mobile users with a small screen device (i.e., 
size of smartphone, not a tablet). Users want results right away, at that moment, and may not be able to spend a lot of 
time to find what they are looking for. 


In order to do mobile rating tasks, you must have experience using a mobile smartphone, which we'll also refer to as a 
mobile phone in these guidelines. If you are not familiar with voice commands, device actions, or phone features, please 
take some time to experiment on a mobile smartphone. For example, you can try some of these voice commands: 


e iPhone Siri voice commands 
e Android voice commands, or say “OK Google” and then “Help” on an Android device 
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12.1 Important Rating Definitions and Ideas 


SERP for [iphone] 


Google 


Should | buy an iPhone? 


Query: [iphone] 
Locale: English (US) 
User Location: New York, NY 


Query: This refers to the word(s) and/or number(s) that a user types or speaks into a mobile phone. In these guidelines, 
queries have square brackets around them. If a user says “navigate home,” we display: [navigate home]. If a user types 
“iPhone” in the search box, we display: [iphone]. 


There are many different types of queries because users ask their phones to do many things, from opening an app to 
calling a friend to searching the web. 


User: The user is the person trying to accomplish something by typing or speaking into a mobile phone with a small 
screen (i.e., size of a smartphone, not a tablet). 


User Intent: When a user types or speaks a query, he or she is trying to accomplish something. We refer to this goal as 
the user intent. 


Locale: All queries have a locale, which is the language and location for the task. Locales are represented by a two-letter 
country code. For a current list of country codes, click here. We sometimes refer to the locale as the task location. 


User Location: This tells us where the user is located. 


Search Engine Results Page (SERP): The page a search engine shows after a user enters a query in the search box. 
The SERP is made up of result blocks. 


Result: We will use the word result to refer to the result block and the landing page. 


e Result Block: This is an individual “block” that appears on the user’s phone in response to the query. The result 
block may display information in the block itself or contain links, or may do both. 


e The Landing Page (LP) is the page you see after you click a link in the result block. 


Device Actions: Mobile phones and other devices can respond to voice commands to perform many actions, such as 
setting an alarm or opening an app. This is a specific type of query that we'll refer to as a Device Action query. 


e Device Action query: Specific type of query where users ask their phone to perform an action. These are 
frequently spoken commands used to complete actions that would normally require interaction with the screen or 
the device's controls. 


e Device Action result: The phone may respond to a Device Action query by performing an action, such as 
calling a phone number, etc. 
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12.2 Understanding the Query 


Understanding the query is the first step in evaluating the task. Remember, a query is what a user types or 
speaks into a mobile phone. 


Some tasks include a query research link, which you should use if you don’t understand the query or user 
intent. Otherwise, please do web research using Google or an online dictionary or encyclopedia. If you still 
don’t understand the query or user intent, please release the task. 


Important: If you research the query on Google, please do not rely on the top results on the SERP. A query may have 
other meanings not represented on Google’s search results pages. Do not assign a high rating to a webpage just 
because it appears at the top of a list of search results on Google. 


Think about users in your locale typing or speaking the following queries into their phone. 


[population of paris], English (US) Find the current population of Paris, France. 


Find the nearest Starbucks location. 
Find weather information in the user location right now. 


Call/dial the number stored for the contact “Mom” on the device. 


12.3 Locale and User Location 


All queries have a task language and task location (referred to in rating tasks as the "Locale"). The locale is important for 
understanding the query and user intent. Users in different locations may have different expectations for the same query. 
Each rating task will show you the User Location information: some tasks have an approximate user location area and 
some tasks have a very specific user location. 


Query: [football] Query: [football] 
Locale: English (UK) Locale: English (US) 


For many or most queries, the user location does not change our understanding of the query and user intent. Here are 
some examples: [facebook.com], [pictures of kittens], [distance between the earth and the moon], [call mom]. If the task 
does not display a user location, please evaluate it as a query where the location does not matter and use your judgment. 


When is the user location important in understanding query interpretation and user intent? Please use both web research 


and your personal judgment to answer this question. Ask yourself, “Would users in one city or country be looking for 
something different than users in another city or country?” 
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12.4 Queries with an Explicit Location 


Sometimes users tell search engines exactly what kinds of results they are looking for by adding the desired location in 
the query, regardless of their user location. We'll call this location inside the query the “explicit location.” The explicit 
location makes queries much easier to understand and interpret. 


Google dallas hotels ra | TS 


Web 


Vit p Deen are - ota. corn _{ My momis coming to 
{> ha on at eB © visit. | need to find a — 
‘hotel for her nearby. =~ 


Query: [Dallas hotels] 
Locale: English (US) 
User Location: Dallas, TX 
Explicit Location: Dallas 


Google new york hotels [a | ss SS e 


- | need a hotel for my trip to 
„O 7~ New York. i 


otel: 
teis 
4 
jew York. NY 
ton New Vork: 5 New York 
tes Square Hotel DD Hiton Midtown 
`~ ee d 
© Hotel Pennsylvania 
Sheraton New York Times Squ. 
sawt 


Query: [New York hotels] 
Locale: English (US) 

User Location: Dallas, TX 
Explicit Location: New York 


tts 


Hotel Pennsylvania 
ttt 


Sometimes the explicit location matches the user location or locale, and sometimes it doesn't. 


When there is an explicit location in the query, pay attention to it! Users use explicit locations to indicate exactly what they 
are looking for. 


12.5 Queries with Multiple Meanings 


Many queries have more than one meaning. For example, the query [apple] might refer to the computer brand or the fruit. 
We will call these possible meanings query interpretations. 


Dominant Interpretation: The dominant interpretation of a query is what most users mean when they type the query. Not 
all queries have a dominant interpretation. The dominant interpretation should be clear to you, especially after doing a 
little web research. 


Common Interpretation: A common interpretation of a query is what many or some users mean when they type a query. 
A query can have multiple common interpretations. 


Minor Interpretations: Sometimes you will find less common interpretations. These are interpretations that few users 
have in mind. We will call these minor interpretations. 
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Query: [apple] 
Locale: English (US) 


Computer company 


Dominant Interpretation 


Query: [mercury] 
Locale: English (US) 


Planet Chemical element 


80 


Mercury 


Common Interpretations 


Common Interpretation 


_ MERCURY / 


Person’s name City 
ie cg \ — “= > \ 
| \ { +... ) 
: A ( : Apple, Oklahoma M \ 
( } C À 
(1 
ig! ig! 
il i 


Minor Interpretations 


Car brand Insurance cane’ Boat engine company 
"N 
y ) Pisce | MERCURY \ 
j \ INSURANCE GROUP j) The Water Calis | 


Minor Interpretations 


12.6 Query Meanings Can Change Over Time 


Remember to think about the query and its current meaning as you are rating. We will assume users are looking for 
current information about a topic, the most recent product model, the most recent occurrence of a recurring event, etc., 


unless otherwise specified by the query. 
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413 president 


User in 1994 


Query: [George Bush] 
Locale: English (US) 


} 


43"4 president 


User in 2004 


Query: [George Bush] 
Locale: English (US) 


| iPhone 1 | iPhone 6 


User in 2007 User in September 2014 


Query: [iphone] 
Locale: English (US) 


Query: [iphone] 
Locale: English (US) 


The interpretation of the query [iphone], English (US) has changed over time as new iPhone models are released. The 

first iPhone was introduced in 2007. Users searching for [iphone], English (US) at that time were looking for the new (at 
the time) first iPhone model. Most users now are looking for the most recent or upcoming iPhone model. In the future, 

new models will come out and the dominant interpretation will change again. 


12.7 Understanding User Intent 


It can be helpful to think of queries as having one or more of the following intents. 


Know query, some of which are Know Simple queries 

Do query, some of which are Device Action queries 

Website query, when the user is looking for a specific website or webpage 

Visit-in-person query, some of which are looking for a specific business or organization, some of which are 
looking for a category of businesses 


12.7.1 Know and Know Simple Queries 


The intent of a Know query is to find information on a topic. Users want to Know more about something. 


Know Simple queries are a special type of Know query. Know Simple queries seek a very specific answer, like a fact, 
diagram, etc. This answer has to be correct and complete, and can be displayed in a relatively small amount of space: 
the size of a mobile phone screen. As a rule of thumb, if most people would agree on a correct answer, and it would fit in 
1-2 sentences or a short list of items, the query can be called a Know Simple query. 
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Know Simple queries may be questions such as [how tall is barack obama]. Frequently, Know Simple queries do not 
have question words. For example, [barack obama height] has the same user intent as [how tall is barack obama], but is 
not in a question format. 


Most queries are not Know Simple queries, such as: 


Broad, complex, and/or in-depth informational queries that do not have a short answer 

Ambiguous or unclear informational queries 

Informational queries on controversial topics 

Informational queries with no definitive “right answer” 

Queries where different users may want different types of information, or different sources of information 


Here are some examples where the Know Simple query asks for a simple fact, which can be answered correctly and 
completely in a small amount of space, and the Know query answer is more complex. 


Know Simple Query Know Query Explanation 


The Know query is a broad information query and different 
[barack obama] users may be looking for different things (e.g., biography, 
books, social media posts, etc.). 


[barack obama height] 
[how tall is obama] 


The Know query is a broad information query and different 
[new york city population 2013] [new york city] users may be looking for different things (e.g., tourist and trip 
planning information, facts, photographs, history). 


The Know query is a broad query for medical information 


[Who is:graves disease nameg [graves disease] and different users may have different needs. There is no 


after] single “answer” for this query. 
The Know query is a broad query for a particular service 
[macy’s store hours] [macy’s gift wrap options] offered by a department store, and does not have a short 


answer. 


[what is the symbol for the 
element nickel] 


The Know query is a broad query and there is no short, 


[what nickel is used for] 
complete answer. 


[who won the 2014 bcs national [who is going to win the bcs The Know query asks for an opinion and there is no 
championship game] national championship game] definitive answer. 


Even though the Know query is theoretically a yes/no 
[what is starbucks stock price] [should i invest in starbucks stock] | question, there is not a single answer that everyone would 
agree on. 


Raters must think about mobile users when deciding if queries are Know Simple. Use your judgment here. 

Important Rating Example: The query [weather] may seem like a broad information query, but most mobile users likely 
have a fairly simple informational need: to find the current or upcoming temperature, and the chance of local weather 
events, such as rain or snow. Therefore, we will consider queries like [weather], [weather today], [weather tomorrow], 
[weather this week] to be Know Simple queries for mobile phone users. 


12.7.2 Do and Device Action Queries 


The intent of a Do query is to accomplish a goal or engage in an activity on a phone. The goal or activity may be to 
download, to buy, to obtain, to be entertained by, or to interact with a website or app. Users want to Do something. 
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Here are some examples. 


Query 

[get candy crush game] 
[online personality test] 
[what is my bmi?] 

[buy citizen kane dvd] 


[bubble spinner] 


Likely User Intent 

Install the Candy Crush game. 
Take an online personality test. 
Calculate BMI (body mass index). 
Purchase this DVD. 


Play Bubble Spinner online or download/open the app. 


Device Action queries are a special kind of Do query. Users are asking their phone to do something for them. Users 
giving Device Action queries may be using phones in the hands-free mode, for example, while in a car. It’s very 
important for mobile phones to accommodate Device Action queries, and we have a high standard for rating these 


results. 


A Device Action query usually has a clear action word and intent. The verb or action word is often at the beginning of the 
query, but a query might start with “OK Google” or “Google” or “Siri” or “I want to.” Use your judgment. 


Here are some examples of Device Action queries. 


Type of Action 
Call someone 


Send a message 


Get map information (e.g., 
directions, traffic, etc.) 


Open an app or webpage, or 
install an app 


Play or view media 


Schedule a meeting 


Set an alarm or timer 


Device Action Query Examples (verb or action shown in bold) 


[call mom’s mobile phone], [place a call to anne jones], [ok google, could you please call my 
mom] 


[send a text to john smith], [text john smith], [sms john smith] 
[navigate to central park], [show me traffic], [show me a map] 


[go to amazon.com], [open facebook app], [open facebook.com], [open the business section of 
the new york times], [siri, would you open facebook for me], [get candy crush app] 


[play me songs from the white album], [watch the life of pi], [show me cute kittens], [i want to 
look at cute kittens], [i want to listen to imagine dragons radioactive] 


[schedule a meeting for 9am] 


[set an alarm for 6am], [alarm for 6am], [countdown timer for 30 secs] 


Here are some examples of queries that are not Device Action queries. 


Query 


[phone number empire state 
building] 


[facebook.com] 


[amazon.com] 


[anne jones] 


12.7.3 Website Queries 


Query is NOT a Device Action Query: Explanation 


There is no action word. The user may be looking for the phone number without the intention of 
calling the number. We'll consider this a Know Simple query. 


There is no action word, such as “open.” We will consider this a Website query. 
There is no action word, such as “open.” We will consider this a Website query. 


There is no action word, such as “call” or “text.” We will consider this a Know query for 
information. 


The intent of a Website query is to locate a specific website or webpage that users have requested. This single webpage 
is called the target of the query. 
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One type of Website query is a URL Query, which can be: 


e Exact, perfectly-formed, working URLs, such as [http:/Awww.ibm.com] or [www.ibm.com] or [ibm.com]. 
e Imperfect URL queries: Queries that look like URL queries, but are not “working URLs”. These URLs do not load 
if you type or paste them into your browser address bar. Even so, we believe users have a specific page in mind. 


Here are some examples. 


[kayak], English (US) View the Kayak website. 
[youtube], English (US) View the YouTube website. 


[ebay], Italian (IT) View the Italian eBay website. 
[new york time health section], English (US) View the Health section of the New York Times website. 
[canon.com eos digital camera], English (US) View the EOS digital cameras page on the Canon website. 


12.7.4 Visit-in-Person Queries and User Location 


Users carry mobile phones with them throughout the day, for example, at work, to school, to restaurants, or running 
errands. One reason to carry a mobile phone is to have help with Visit-in-Person queries, such as finding coffee shops, 
gas stations, ATMs, restaurants, etc. Because mobile phones are often used for Visit-in-Person queries, make sure to 
consider visit-in-person intent as a possibility for mobile phone users. 


Some queries clearly “ask” for nearby information or nearby results (e.g., businesses, organizations, other nearby places). 
Some queries are not asking for nearby information or nearby results. Here are some examples. 


Queries with Visit-in-Person Intent Query with Non-Visit-in-Person Intent 


Google chinese restaurants nar A ~ Google how tall is tom cruise 


s Web 


| want to get Chinese : = (1.70 m) 
food for dinner. Ton Cries, Hohl j 
_ How tall is Tom Cruise? 


Kirin Chinese Restaurant aw 
recs ( 


China Wok Restaurant 
29 ek 


Hangen Szechuan 
34 keke 


Query: [chinese restaurant] 
7 Photos Of Tom Cruise Pretending To Be 
Locale: English (US) Taller Than His Co-Stars 


je O ASEE Query: [how tall is tom cruise] 


Ap 
vever, with the power of technology he is able to 


be tilar amn enone be Locale: English (US) 


Google gas stations 
i aa Google atest B 

Web eT 

- Wikipecia, the ree wo 

IQTest.com 

wc latest con 

Click here to take our ree private, and fun 1A test 

Our originat 1A test is tne most scientifically valid 1a 


— I would like to take an 


| need to find a gas 
g = IQ test online. 


station right away! ~ i Aone 
ye IQ Test 
E) Test my 1a is designed to test 
Pg 1O. Questions we use are mainly 
EER logical 
dekrete for 16700 user 
more Google Play apps for iq test 
Chevron Free IQ Test - Fast, Free and Accurate 


Online IQ Test 
www free-iqtest net 


Query: [gas stations] Peat gee meuniere Query: [ig test] 
Locale: English (US) Resa as cans ah Locale: English (US) 


AAAA 17 reviews 


More examples: [boston red sox], [washington post], [scrabble 
cheat], [definition of sedentary], [aapl], [beyonce], [angry birds], 
[small dog breeds], [dance videos], [oscars 2012], [pick up lines], 
[bank of america login] 


More examples: [pizza], [yoga class], [coffee shops], [movie 
showtimes], [car repair], [dentists], [bank of america atm 
locations], [starbucks near me] 
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And some queries could go either way. Some users may want nearby results and others may not. 


aD mW © @ Bp 


Shop by Department 


Rollbacks 


Find a Store 


Your Orders 


Weekly Ad 


Pharmacy 


za 
a | want to go toa 
„o (~~ Walmart near me. 


Query: [walmart] 
Locale: English (US) 


( 


| want to shop online on 


{ 


~O7> the Walmart website. _ 


Query: [walmart] 
Locale: English (US) 


Here are some examples of queries with both visit-in-person and non-visit-in-person intent: 


[hotels] 

[post office] 
[apple store] 
[citibank] 

[best buy] 

[office depot] 
[target] 

[library] 

[bank of america] 


[the gap] 
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Sometimes, the user location can change our understanding of the query. For users close to Sunnyvale, California, the 
query [turmeric] could have two different interpretations: a popular restaurant named Turmeric or the spice turmeric. 


Google turmeric restaurant sun B 


Web s News re~ 


ENE < Whatis the address of 
ae | >— Turmeric? i 


Dishdash (ri 


Turmeric Restaurant E 
36 kkkt 


Query: [turmeric] 
2 ess: | eee Locale: English (US) 
rant com User Location: Sunnyvale, CA 


Turmeric 
a. 


| want to learn about 
— the spice turmeric. > _ 


Query: [turmeric] 
Locale: English (US) 
User Location: Sunnyvale, CA 


In most other user locations, there is no restaurant (or anything else) named Turmeric and there is just one interpretation 
of the query [turmeric]: the spice. The Sunnyvale restaurant is not well-known outside of Sunnyvale, California. 


| want to learn about 
-O © the spice turmeric. 


Query: [turmeric] 
Locale: English (US) 
User Location: Lincoln, NE 


Use your common sense when thinking about queries and whether they have possible visit-in-person intent. 
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12.7.5 Queries with Multiple User Intents 


Many queries have more than one likely user intent. Please use your judgment when trying to decide if one intent is more 
likely than another intent. 


[harvard] Depending on the user need and location, users may want to visit the official homepage (Website), 


get directions (Visit-in-Person), or learn more about the school (Know). 


Most users want to go to a nearby Walmart (Visit-in-Person) or view the homepage to shop online 
[walmart] à é : 
(Website). Some or few users may want to learn more information about the company (Know). 


12.8 Understanding Result Blocks 


The following sections contain examples of different types of queries and results. In these guidelines, please 
assume that the result blocks and pages are easy to use on the phone, unless otherwise noted. 


12.8.1 Web Search Result Block Examples 


Web Search Result Blocks typically have a title link, a URL and a “snippet” of text describing the page. For many queries, 
Web Search Result Blocks are the most helpful type of result. 


Query, User Location, User Intent 


Web Search Result Block 


Query: [cuisinart food processor reviews] 
User Location: Oklahoma City, Oklahoma 


User Intent: This is a Know query. The user 
wants to find recent reviews of Cuisinart food 
processors. 


Result: This is a Web Search Result Block 
that has a link to a landing page with Cuisinart 
reviews. 


Query 


Cuisinart food a 
processor reviews 


Cuisinart Elite Die Cast Food 
Processor, 12-Cup | Williams- 
Sonoma 
www.williams-sonoma.con/.../cuisinart-e... 


kkkk& Rating: 4 1 review 49.9 
... larger capacity, this kitchen workhorse 
represents the next generation of Cuisinart 
food processors. ... Read Reviews ... 


Web Search 
Result Block 


Query: [broadway tickets] 
User Location: New York City, New York 


User Intent: This is a Know query or Do 
query. The user wants to search prices and/or 
purchase tickets to a Broadway show in New 
York City. 


broadway tickets 


Broadway Tickets | Broadway Shows | 
Theater Tickets | Broadway.com 
www.broadway.com/ 

The most comprehensive source for Broadway 
Shows, Broadway Tickets, Off- Broadway, London 
theater information, ... 

Shows - Broadway Buzz - The Lion King - Wicked 
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12.8.2 Special Content Result Block Examples 


Special Content Result Blocks (SCRBs) appear in the search results page, along with Web Search Result Blocks. They 


are frequently, but not always, the first result on the search results page. 


Special Content Result Blocks are designed to show content directly to users on the search results page. From working 
calculators to playable videos to interactive weather information, these results help users immediately get information or 
content, and may not require additional “clicks” or page loads. 


Because mobile phones can be difficult to use, SCRBs can help mobile phone users accomplish their tasks very quickly, 


especially for certain Know Simple, Visit-in-Person, and Do queries. 


Important: Please assume that any interactive features work and function properly. Some notes: 


e Allresult blocks are “screenshots” or images of search results with prominent links enabled. Unfortunately, a 
screenshot or image of an interactive result block will not function as it would for a real user. For the purpose of 
rating, please assume that interactive result blocks do function as intended. Try to interact with the result 


block as some links, buttons, or other features may work. 


e There may be a delay between when the rating task is created and when you actually rate the block, causing 
some information in special content result blocks to be a few hours or even days out of date. Stock price or 
weather informational blocks are designed to give users extremely current and timely information. However, due 
to a delay in rating time, the information may no longer be accurate. Don’t penalize a special content result block 
for being out of date. Assume that the blocks show current information for users, unless instructed 


otherwise. 


Query, User Location, User Intent 


Query: [weather] 
User Location: Chicago, Illinois 


User Intent: This is a Know Simple query 
since mobile users have a fairly simple 
informational need: find the current 
temperature and chance of rain or snow. The 
user wants to know the weather for the User 
Location. 


Note: Assume the block shows current 
information for users. 


Special Content Result Block 


Chicago, IL 


a63 
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Query, User Location, User Intent Special Content Result Block 


r . Query 
Query: [emma stone movies] 


User Location: Des Moines, lowa 


Emma Stone > M 


User Intent: This is a Know query. The user 
wants to get information on movies with Emma 
Stone. 


Special Content Result 
Block 


A ; ; 
Result: In this result block, users can 7. Ss me 
immediately see some popular movies starring 
Emma Stone, with an option to click on the 
links to learn more about each movie. Users me The nelp 


. g P Amazing Spi... 2011 
can also swipe to see a list of more movies. p 


Note: users can swipe left 
to see a list of more movies 


ac Query 


Query: [calories in a banana] 


User Location: Oakland, California 

Special Content 
a : Bananas Result Block 

User Intent: This is a Know Simple query. 


The user wants to find out how many calories 
are in a banana. 


1 medium (7" to 7-7/8" long) (11£ $ 


Note: you can click the 
arrow to expand the 
block and see more 
information 


Major League Baseball 
Query: [baseball scores] Scores & Schedule 


User Location: Wichita, Kansas 


Special Content 


User Intent: This is a Know query. The user Result Block 


wants to find the most recent Major League 
Baseball scores. 
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Query, User Location, User Intent 


Query: [coldplay fix you video] 

User Location: New Orleans, Louisiana 
User Intent: This is a Do query. The user 
wants to watch the music video for the song 
“Fix You” by Coldplay. 

Result: By clicking on this result block, users 


can play the music video for the song on their 
phone, learn more about the artist/album, etc. 


Query: [what is the tallest tree] 
User Location: Boise, Idaho 
User Intent: This is a Know Simple query. 


The user wants to know what type of tree is 
the tallest. 


12.8.3 Device Action Result Block Examples 


Special Content Result Block 


coldplay fix you video 


ALL VIDEOS IMAGES NEWS MAPS 


Fix You 
Song by Coldplay 


v 


© Coldplay - Fix You 
4:56 - YouTube 


Artist: Coldplay 
Album: X&Y 


$ what is the tallest tree 


redwood 


Hyperion is the name of a coast 
redwood (Sequoia sempervirens) in 
Northern California that was 
measured at 115.61 metres (379.3 
ft), which ranks it as the world's 
tallest Known living tree. 

Hyperion (tree) - Wikipedia, the free 
encyclopedia 
en.wikipedia.org/wiki/Hyperion_(tree) 


Feedback 


A mobile phone should respond to a Device Action query and do what the user is asking. If the block is a Device Action 
query to open an app, please assume that the user has the app installed on their phone. If the query is to download the 
app, please assume that the user does not have the app installed on their phone. 


Query, User Location, User Intent, Result 

Query: [open angry birds] 

User Location: Little Rock, Arkansas 

User Intent: This is a Device Action query. 

The user wants the device to open the Angry 


Birds app so he/she can play the game. 


Result: In this result block, users can click the 
link to open the app. 


B. Open app 


Device Action Result Block 


a Query 


open Angry Birds 


B 


Angry Birds 


Device Action 


= | Result Block 


Copyright 2017 75 


Query, User Location, User Intent, Result Device Action Result Block 


Query: [go to amazon.com] 

User Location: Glendale California 

User Intent: This is a Device Action query. 
The user wants to go to amazon.com to 


interact with the website. 


Result: In this result block, users can click 
the links to go directly to the website. 


Query: [set alarm for 5 o’clock am] 
User Location: Tacoma, Washington 


User Intent: This is a Device Action query. 


The user wants to set the device’s alarm to go 


off at 5:00 AM. 


Result: In this result block, the query has 
initiated the alarm on the phone to be set for 
5:00 AM. 


Query: [send text to mom] 
User Location: Tacoma, Washington 


User Intent: This is a Device Action query. 
The user wants to send a text message to a 
contact nicknamed “Mom.” 


Result: In this Device Action Result Block, 
the query has initiated a text to the contact 
nicknamed “Mom,” with an option to text 
either the work or mobile number. 


ao Query 


go to amazon.com 


Amazon.com: Online Shopping for 
Electronics, Apparel, Computers ... 


www.amazon.com 


—————— Result Block 
9 Eemme|— X 


Device Action 


y set alarm for 5 o'clock am oO 


Setting alarm... 


5:00 AM 


©® Set alarm 


g send text to mom 


Work or Mobile? 


Mom 


Work 


Mobile 


Cancel 
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Query, User Location, User Intent, Result Device Action Result Block 


call Best Buy © 
Query: [call best buy] 


User Location: Mountain View, California 


User Intent: This is a Device Action query. 
The user wants to call the nearest Best Buy 
store. 


ere Í 
14 ooge ETS 


Result: In this Device Action Result Block, the pore ine N al 


query has initiated a call to the nearest Best 
Buy location. = 
R Call 


12.8.4 How Device Action Results are Displayed in Rating Tasks 


Important note: Users issuing queries actually experience the phone’s response to the query, while raters are given a 
description of it. If an iPhone user says “Siri, call Mom,” the iPhone displays a visual prompt to show that it’s dialing a 
phone number. In these rating tasks, you will see a description of this action. 


Please also assume that the phone successfully performs the action in a fully satisfying way. 


Below are some additional examples of Device Action queries, along with the corresponding “action text” displayed in the 
rating task. 


This is what ihe user sees on his orher phone forthe This is what you (the rater) might see in the rating task 
Device Action query 


set alarm for 30 
minutes 


The request has triggered the following action: 
SS 

Query Time: 2013-07-18 

Action: 


@® Set alarm 


The left side shows what the user sees on the phone for the query [set alarm for 30 mins], while the right side shows the 
“action text” that you (the rater) might see in the rating task. The text on the right indicates that the user issued the 
query at 1:48 PM (13:48:00), and the device has set its alarm to go off 30 minutes later at 2:18 PM (14:18:00). 
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dist vijat mo Wid get E a e This is what you (the rater) might see in the rating task 
Device Action query 


open facebook app 


. The request has triggered the following action: 
Opening app... Al 2 ag 9 


Action: Open an app 


Facebook f 


The left side shows what the user sees on the phone for the query [open facebook app]. In this case, the phone shows 
an “Opening app” message and then opens the app. The right side shows the “action text” that you (the rater) might see 
in the rating task. The text indicates that the device responded by opening the Facebook app on the user’s phone. 


The request has triggered the following action: 


= Action: Make a call 
Contact name: "Dan" 


Calling ... 


Mobile 


& call 


The left side shows what the user sees on the phone for the query [call dan], while the right side shows the “action text” 
that you (the rater) might see in the rating task. The user wants to dial the number stored for the contact “Dan” on the 
device. This text indicates that the device responded by displaying the contact’s phone number and dialing the number. 
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This is what the user sees on his or her phone for the 


Device Action query This is what you (the rater) might see in the rating task 


call target 


p Menlo Park foe: 
Zp Palo Alto } 
* Menlo Park K 
ntor 
a 


Query Time: 2013-08-20 14:19:19 
Action: Make a call to a business or an organization 
BU e Name. targe 


Location: 555 Showers Drive, Mountain View, CA 94040 
Phone number: (650) 965-7764 


Z Make a Call mi a business or an organization 


Location: 298 West McKinley Avenue, Sunnyvale, CA 94086 
Phone number: (408) 702-1012 


Action: Make a call fo a business or an organization 
Bu 


Location: 210745 ‘Stevens Creek Boulevard, Cupertino, CA 95014 
Phone number: (408) 725-2651 


For some Device Action queries, the device displays a list of options to choose from before it can respond to the 
Device Action query. Here is an example for the query [call target]. The user is shown a list of Target stores in the 
area to choose from. This example shows that the Device Action result taken by the device will be to call the desired 
Target store after the user makes his/her selection. 


play adele 


Name: Adele 
Action: Play media 


D 4 Play Music 


Here is an example for the query [play adele]. The user is shown a "Play media" type of result. In this case, since the 
user wants to play music, the Device Action result shown is a "Play Music" button to click. Clicking this button will play 
the specified song if the user has the song on his/her device. If not, clicking the result box will give the user options to 
hear the specified song online. 
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12.9 Rating on Your Phone Issues 


We understand that raters using different phones, operating systems, and browsers may have different experiences. In 
general, do what you would do naturally, and rate based on your experience. Here are some additional points to keep in 
mind when you are rating on your phone: 


e |f you see one of these messages when you open a landing page on your phone: 


o Asks whether you want to open the page in the browser or the website’s app: select and evaluate the 
webpage. However, if the result automatically opens an app by default based on your phone’s settings 
(e.g. some users have set all YouTube pages to automatically open the result in the YouTube app), you 
should rate your natural app experience—you do not have to change your phone’s default settings. 

o Asks whether you want to visit the mobile page or desktop page: it is fine to select the mobile page as 
long as the landing page is the same. Sometimes, the mobile option will bring you to the mobile 
homepage instead of the specific URL in the task. You may need to check that the mobile page is in fact 
the same URL as the desktop page. 


e Please open and look at PDF files. Your experience may be easier or harder than other users depending on your 
phone and browser, but you should still open the PDF file and look at it. 


e Occasionally, you may be assigned some of these rating tasks on a desktop computer, but please rate from the 
perspective of a mobile user unless otherwise instructed. 


e You should assume queries were issued on a smartphone. For some queries, there are results that are helpful for 
specific types of phones—for example, some apps will only work on an Android or iPhone. Please assume that 
queries with device-specific results were issued on an Android device, unless explicitly stated otherwise in the 
instructions. 
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Part 3: Needs Met Rating Guideline 


13.0 Rating Using the Needs Met Scale 


There are many different kinds of queries and results, but the process of rating is the same: Needs Met rating tasks ask 
you to focus on mobile user needs and think about how helpful and satisfying the result is for the mobile users. 


Needs Met Rating 


i—__ —_+—_$ t+ —_$i—_—_+—_ + —_ + 


N/A FailsM SM MM HM FullyM 


This is what the Needs Met rating slider looks like: 


Porn: No Foreign Language: No, Did Not Load: No  Upsetting-Offensive: No 


Rating |Deseription o 


A special rating category, which only applies to certain queries and results. All or almost all 
Fully Meets (Fully M) mobile users would be immediately and fully satisfied by the result and would not need to 
view other results to satisfy their need. 


Highly Meets (HM) Very helpful for many or most mobile users. Some users may wish to see additional results. 


Helpful for many users OR very helpful for some mobile users. Some or many users may 
Moderately Meets (MM) wish to see additional results. 


Helpful for fewer mobile users. There is a connection between the query and the result, but 
Slightly Meets (SM) not a strong or satisfying connection. Many or most users would wish to see additional 
results. 


r f Completely fails to meet the needs of the mobile users. All or almost all users would wish to 
palig 1o Meet (FANEM) see additional results. 


Please note that you may assign in-between ratings. Use in-between ratings if you think the rating of a result falls 
between two labels. You can either drag the slider or click on the point that you want the slider to land on. 


13.1 Rating Result Blocks: Block Content and Landing Pages 


For Needs Met rating, you will assign a rating to each result. Each result includes the content inside the result block and 
landing pages associated with the result. 


Which part of the result do you rate? It depends on both the query and the result block: 


Type of Block What to Rate 


The content inside this type of block should always play a large role in your rating. 
Special Content Result Block 
(SCRB) In addition, think about whether a user would click on the link(s) if available, in order 
to satisfy their user need. 
Note: Assume that interactive 


result blocks function as intended. e |f most users would not click, rate the Special Content Result Block based 

Try to interact with the result block on the block content alone. 

as some links, buttons, or other e |f some or many users would click, you may consider the helpfulness of the 

features in your rating task may landing page(s) in addition to the content in the block. In this case, both 

work. need to be helpful to justify a high rating. If either the content in the block 
or on the landing page is unhelpful, give a low rating. 

Web Search Result Block A click is required, and you should evaluate the landing page to assign a rating. 

Device Action Result Block Base your rating on the helpfulness of the action itself. 
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For example, think about the query [what does love mean]. 


Result Block 


Special Content Result Block 


Web Search Result Block 


Special Content Result Block 


love 
Aav 4% 


noun 


1. an intense feeling of deep affection. 
"babies fill parents with intense feelings of love” 
s: deep affection, fondness, tenderness 
warmth, intimacy, attachment 
endearment; More 


2. a person or thing that one loves. 
s as the love of his life" 
synonyms: beloved, loved one, love of one’s life. 
dear, dearest, dear one, darling 
sweetheart, sweet, angel, honey: More 


verb 


1. feel a deep romantic or sexual attachment to 
(someone). 
"do you love me?" 
synonyms: care very much for, feel deep affection 
for, he dear, adore, think the world 
of, be devoted to, dote on, idolize. 
worship; More 


Translations and more definitions 


What Is Love? We Spoke to the Merriam- 
Webster Lexicographer ... 
www.themarysue.com/definition-of-love/ 


Mobile-friendly - Nov 7, 2013 - “Love” is a word we all 


use and understand, but defining such a complex 
idea seems like a daunting ... 


Love is a Choice. Do you ever wonder 
what it means to really love someone? 
We're bombarded today with movies 
and videos and romance novels that 
give us a false image of what it means 
to love. 

What does it mean to really love someone? 
- Living on the Edge 
livingontheedge.org/.../what-does-it-mea.. 
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Most users would probably not click on 
the Special Content Result Block, 
because the block contains a large 
amount of helpful content and has no 
obvious landing page link associated 
with it. 


Therefore, base your rating on the 
content inside the block itself. 


In this case, users would have to click on 
the web search result in order to get an 
answer to the question. 


Therefore, base your rating on the 
content of the landing page. 


While the block content may be helpful, 
this question may require a more in 
depth answer for some users. Some or 
many users might click on the link, in 
addition to looking at the content in the 
block itself. 


Therefore, base your rating on both the 
block and the landing page. 
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Here are some examples of Special Content Result Blocks where the block should be rated primarily on the content inside 
the block itself. As always, please use your judgment. 


Query: [utopia animal hospital] 


Court Ave 


N Claybre 


ICAL 
RICT Court Ave 


Madison Ave 9 


Madison Ave 


Monroe 
Utopia Animal Hospital 
4.5 kkk KK 18 reviews 


Animal Hospital 

1157 Madison Avenue, Memphis, TN 38104 
(901) 746-8758 

» Closing soon - 7:30 am — 5:30 pm 


N Y © 


Call Directions Website 


More about Utopia Animal Hospital 


Query: [how to find security code on 
visa] 


The card security code 

is located on the back of 
MasterCard, Visa and 

Discover credit or debit 

cards and is typically a separate 
group of 3 digits to the right of the 
signature strip. On American 
Express cards, the card security 
code is a printed, not embossed, 
group of four digits on the front 
towards the right. 


Card security code - Wikipedia, the free 
encyclopedia 
en.m.wikipedia.org/.../Card_security_cod.. 


Feedback 


Query and Special Content Result Block 


Query: [chicago weather] 


Chicago, IL 
Tue, 5 PM, Mostly Cloudy 


a10" 


] ' | 1 ' 
7PM 12 AM 5AM 10 AM 3PM 


Precip: 10% 
Humidity: 75% 
Wind: 4 mph 


THU FRI SAT SUN 


2 2 


27° 32° 
14 19° 


TUE WED 


aa 2 


Query: [how did Lincoln die] 


Assassination 


Abraham Lincoln, Cause of death 


“How Did Lincoln Die? Everyone knows that the 
ball John Wilkes Booth fired into Abraham 
Lincoln's brain inflicted a terrible, mortal wound. 
But when a prominent neurosurgeon began to 
investigate the assassination, he discovered 
persuasive evidence that Lincoln's doctors must 
share the blame with Booth's derringer.” 


How Did Lincoln Die? | American History Lives 
at American Heritage 
American Heritage > content » how-did-l... 


More about Abraham Lincoln 
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Query: [calories in a banana] 


105 calories 


Bananas 


1 medium (7" to 7-7/8" long) (1189) + 


More about Banana 


Query: [movies san francisco] 


Movies near San Francisco 


Genre 
All Genres ~ 


San Andreas Entourage 


Vv 


More movies 
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13.2 Fully Meets (FullyM) 
Fully Meets is a special rating category, which can be used in the following situations: 


e The query and user need must be specific, clear, and unambiguous. 

e The result must be fully satisfying for mobile users, requiring minimal effort for users to immediately get or use 
what they are looking for. 

e All or almost all users would be completely satisfied by the result—users issuing that query would not need 
additional results to fully satisfy the user intent. 


In other words, the Fully Meets rating should be reserved for results that are the “complete and perfect response or 
answer” so that no other results are necessary for all or almost all users to be fully satisfied. 


You will need to use your judgment to decide whether a result block can Fully Meet the user need. Here are some 
scenarios when the Fully Meets rating is appropriate: 


e The user is clearly looking for a specific webpage or website and the result block with the specific webpage or 
website Fully Meets the user’s need. 

e The user is trying to complete a device action and the result block accomplishes the desired request. 

e The user is looking for a very specific fact or piece of information and the result block provides the information 
immediately, thoroughly, accurately, and clearly. No other results would be needed. Before using the Fully Meets 
rating for queries seeking a very specific fact or piece of information, you must check for accuracy and confirm 
that the information is supported by expert consensus where such consensus exists. 


Fully Meets may apply in other situations as well. Ask yourself whether the result block alone is the perfect and complete 
result that would fully satisfy all or almost all users. Be conservative when using the Fully Meets rating. When in doubt, 
consider a lower rating. 


Note: If a result block is very close to being fully satisfying, but the block alone may not be sufficient in order to fully satisfy 
all or almost all users, a rating of Highly Meets+ may be appropriate. 


Sometimes you have to think about how specific the user need is and will need to make a judgment call on whether it is 


specific enough to possibly have a Fully Meets rating. 


13.2.1 Examples of Fully Meets (FullyM) Result Blocks 


Query and User Intent Result Rating Fully Meets Explanation 


' ' | | | 
FailsM SM MM HM FullyM 


What you might see in the 


Query: [amazon.com] = amazon.com rating task: 


User Location: Action: Open a specific The query has clear intent to go 
Austin, Texas Opening website © to amazon.com and the phone 
webpage : : i ; 

Amazon.com: Online Shopping for Electro... is opening the specified website. 

User Intent: Go to 


amazon.com. g 


URL: http:/Awww.amazon.com/ 


Website Name: Amazon.com: 
Online Shopping for 
Electronics, Apparel, 
Computers ... 
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Query: [amazon] 


User Location: 
Austin, Texas 


User Intent: Go to the 
Amazon website. 


Query: [target website] 


User Location: Jacksonville, 
Florida 


User Intent: Go to the Target 
website. 


Query: [shop nordstrom 
online] 


User Location: Monterey, 
California 


User Intent: Go to the 
Nordstrom website for online 
shopping. 


Query: [titanic imdb] 


User Location: Lexington, 
Kentucky 


User Intent: Go to the IMDb 
page for Titanic. 


Query: [www.yahoo.cOm] 


User Location: Denver, 
Colorado 


User Intent: Go to the Yahoo 
website. 


Query: [cnn health] 


User Location: Annapolis, 
Maryland 


User Intent: Go to the Health 
section of cnn.com. 


Query: [cnn] 


User Location: Annapolis, 
Maryland 


User Intent: Go to the CNN 
website. 


Amazon.com 
www.amazon.com 


Mobile-friendly - Online retailer of books, movies,| 35M 


music and games along with electronics, toys, 
apparel, sports, tools, groceries and ... 


Target : Expect More. Pay Less. 
www.target.com/ 

Spend $50 and get free shipping on over 
500K items. Choose from a wide selection 
of furniture, kids ... 


Mountain View Target Find Store 
Women's Electronics 


Home : Decor, Furn...Weekly Ad 


Nordstrom 

m.shop.nordstrom.com/ 

Mobile-friendly - Shop online for shoes, clothing, FailsM 
jewelry, dresses, makeup and more from top 

brands. Make returns in store or by mail. 


Titanic - IMDb 
m.imdb.com/title/tt0120338/ 

kk kK Rating: 7.7/10 - 663,660 votes 
Mobile-friendly - Titanic -- Experience James 
Cameron's Titanic like never before. Leonardo 
DiCaprio and Kate Winslet Titanic -- Jack ... 


FailsM 


Yahoo 

https://www.yahoo.com/ 

Mobile-friendly - Anew welcome to Yahoo. The 
new Yahoo experience makes it easier to discove 
the news and information that you . 


FailsM 


Health News - Medical, 
Mental and Dental ... 
www.enn.com/HEAL TH/ 

Mobile-friendly - Discover the latest news and 
video about health at CNN. CNN covers diet, 
fitness , parenting, conditions and more, .. 


CNN.com 

www.enn.com/ 

Mobile-friendly - CNN.com delivers the latest 
breaking news and information on the latest top 
stories, weather, business, entertainment, ... 


FailsM 
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The query has clear intent to go 
to the amazon.com website. 
While there may be other 
interpretations for the query, the 
dominant one is by far the 
website. 


The query has clear intent to go 
to target.com. 


The query has clear intent to go 
to the Nordstrom website. 


The query has clear intent to go 
to the IMDb page for Titanic. 


Even though this is an imperfect 
URL query, it’s clear the user 
wants to go to the Yahoo 
website. 


The query has clear intent to go 
to the Health section of the 
cnn.com webpage. 


The query has clear intent to 
access CNN news and content. 
It is not clear whether the user 
has the app installed, but the 
website has the content the user 
is searching for, and fully 
satisfies the user intent. 
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Query: [yelp] 


User Location: Raleigh, 
North Carolina 


User Intent: Open the Yelp 
app or go to the website at 
yelp.com. 


Query: [lebron james stats 
basketball- reference.com] 


User Location: 
Miami, Florida 


User Intent: Go to the player 
statistics page for LeBron 
James on a specific website. 


Query: [open instagram] 


User Location: 
Laguna Beach, California 


User Intent: Open the 
Instagram app on the user’s 
device. 


Query: [open bing search 
app] 


User Location: Dallas, Texas 
User Intent: Open the Bing 


Search app on the user’s 
device. 


Yelp 
Yelp app - Installed 
>s People use Yelp to search for everything 
from the city's tastiest ... 
Open on yelp.com 


LeBron James NBA & ABA Stats | 

Basketball-Reference.com 

www.basketball-reference.com > ... > J 

6/19 Factory of Sadness: Intel: LeBron } ' I 
James Opting in or Signing With Cavs: Just FailsM Fully 
following the NBA Finals we wrote that 

Well, that's the latest talk, as noted by 

ESPN and others. 


What the user sees: i 


FailsM 


S open Instagram 


Opening app... 


What you might see in the 
rating task: 


Action: Open an app 


Instagram App: Instagram 


Bing Search 

With this app, you can save money 
with Bing Rewards, keep up with 
trending stories and images, and se. 
... Start with the search box and click on a result 
to open it up in a built-in browser, or go straight ... 
Ak kxk * from 38,657 users 


l> 


FailsM FullyM 


he View on Google Play (free) 
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This result Fully Meets the user 
need whether they want the Yelp 
app or the Yelp website, which 
are both popular. 


Note: If you see this kind of 
result, assume that the user has 
the app installed on their phone. 


Clicking on the Yelp link above 
the logo opens the app directly, 
and clicking on the “Open on 
yelp.com” link gives the option of 
opening the app or website. 


opens Yelp app directly 


Yelp app - Installed 
b s People use Yelp to search for everything 


from the city's tastiest ... 
Open on yelp.com 
gives option of opening 
app or website 


The query has clear intent to go 
to the player statistics page for 
LeBron James on a specific 
website, 
basketball-reference.com. 


This Device Action query is 
clear and specific: the user 
wants to open the Instagram 
app. The result block shows the 
phone in the process of opening 
the specified app—it Fully 
Meets the user need. 


The result block shows that the 
phone has detected that the app 
is not installed and gives the 
option of downloading the app. 
This result Fully Meets the user 
need. 


Note: Remember that you 
should assume that queries 
were issued on an Android 
device when rating Device 
Action Result Blocks, unless 
explicitly stated otherwise in the 
instructions. Therefore, the 
Google Play result for this query 
is appropriate. 
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Query: [open facebook app] 


User Location: Richmond, 
Virginia 


User Intent: Open the 
Facebook app on the user’s 
device. 


Query: [set alarm for 5 
o'clock am] 


User Location: Tacoma, 
Washington 


User Intent: Set the device’s 
alarm to go off at 5:00 AM. 


Query: [chef chu phone 
number] 


User Location: Los Altos, 
California 


User Intent: Find the phone 
number for the restaurant 
called Chef Chu’s. 


Query: [call best buy] 


User Location: Mountain 
View, California 


User Intent: Call the nearest 
Best Buy store. Note that 
there is only one Best Buy 
store located in Mountain 
View. 


What the user sees: 


f Open Facebook 


What the user sees: 


set alarm for 5 o'clock am o) 
Setting alarm... 


5:00 AM 


W Set alarm 


San Antonio g 
Shopping Center 
Je. Walmart & 
oy 


k4 
© 

fe, 

9; 


t (650) 948-2696 


Chef Chu's, Phone 


More about Chef Chu's 


What the user sees: 


& call Best Buy 


Best Buy - Mountain View 
st Charle ston Road, Mountain 


ae oa 


What you might see in the 
rating task: 


Action: Open an app 


App: Facebook 


FailsM FullyM 


What you might see in the 
rating task: 


Action: Set alarm 


Alarm Time: 5:00 AM 


} | | 
FailsM FullyM 


ja 1 Í | 
FailsM FullyM 


What you might see in the 
rating task: 


Action: Make a call 


Business Name: Best Buy 


Location: 2460 E Charleston Rd, 


Mountain View, CA 94043 


Phone number: (650) 903-0591 
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This Device Action query is 
clear and specific: the user 
wants to open the Facebook 
app. The result block shows the 
phone’s confirmation that it is 
opening the app—it Fully Meets 
the user need. 


This Device Action query is 
clear and specific: the user 
wants to set the phone’s alarm 
clock for 5:00 AM. The result 
block shows the phone in the 
process of setting the alarm at 
the specified time—it Fully 
Meets the user need. 


Chef Chu’s is a Chinese 
restaurant located in the user 
location. This result block 
immediately shows the correct 
phone number with the option to 
call the number directly on the 
device—it Fully Meets the user 
intent. 


Note: You must check for 
accuracy before using the Fully 
Meets rating. You can verify 
that the phone number in this 
SCRB is correct by checking 
Chef Chu's official website. 


This Device Action query is 
clear and specific: the user 
wants to call the nearest Best 
Buy store. The result block 
shows the phone in the process 
of calling the store with the 
correct phone number—it Fully 
Meets the user need. 


Note: You must check for 
accuracy before using the Fully 
Meets rating. You can verify 
that the phone number in this 
Device Action block is correct by 
checking Best Buy's official 
website for the location in 
Mountain View. 
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Query: [open map of italy] 


User Location: Baltimore, 
Maryland 


User Intent: Open a map of 
Italy on the user’s device. 


Query: [navigate to yosemite 
national park] 


User Location: Seattle, 
Washington 


User Intent: Navigate to 
Yosemite National Park from 
the user’s location. 


Query: [how to find security 
code on visa] 


User Location: Chicago, 
Illinois 


User Intent: Find out how to 
locate the security code ona 
Visa card. 


What the user sees: 


open map of Italy 


as 
Israel, 05T 


Sgal = 
ap tec, Scope 


Italy 
Italy 


9 Open map 


What the user sees: 


navigate to Yosemite 
National Park 


Driving 


A Yosemite National Park 
‘osemite Village, C 5389 


Å Navigate 


The card security code 
is located on the back of 
MasterCard, Visa and 
Discover credit or debit 
cards and is typically a separate 
group of 3 digits to the right of the 
signature strip. On American 
Express cards, the card security 
code is a printed, not embossed, 
group of four digits on the front 
towards the right. 

Card security code - Wikipedia, the free 
encyclopedia 
en.m.wikipedia.org/.../Card_security_cod.. 


Feedback 


What you might see in the 
rating task: 


Action: Open map 


Location: Italy 


FullyM 


What you might see in the 
rating task: 


Action: Navigation 


Destination: Yosemite National 
Park 


Destination Address: Yosemite 
Village, CA 95389 
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This Device Action query is 
clear and specific: the user 
wants to see a map of Italy. The 
result block shows a map with a 
link to open it in Google 
Maps—it Fully Meets the user 
need. 


This Device Action query is 
clear and specific: the user 
wants to navigate to Yosemite 
National Park. The result block 
shows the phone in the process 
of opening up the Navigation 
app, which will give the user 
turn-by-turn directions from the 
user’s location—it Fully Meets 
the user need. 


The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users 
on mobile devices to read. The 
block also includes an image 
showing users exactly where to 
find the code, and the block 
Fully Meets the user need. 
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Query and User Intent 


Query: [starbucks stock 
price] 


User Location: Alexandria, 
Virginia 


User Intent: Find the current 
stock price for Starbucks. 


Query: [new york city 
population 2012] 


User Location: Pasadena, 
California 


User Intent: Find the 
population of New York City 
back in 2012. 


Query: [what is the weather 
forecast for today] 


User Location: 
Mountain View, California 


User Intent: Find weather 
forecast information for the 
current day in Mountain View, 
California. 


Query: [london time] 


User Location: New York, 
New York 


User Intent: Find the current 
time in London. 


Result 


Rating 
ETA 


92.31 +0.13 (0.14%) 


d 1 3m y Sy max 


i ' ' ' | 


Man, FailsM SM MM HM FullyM 


[8.337 million (2012) 


ee 
j i \ i i 
FailsM SM MM HM FullyM 
Los Angeles Chicago 
3.858 million (2012) 2.715 million (2012) 
Mountain View, CA 
Thu, 8 AM, Overcast 
50 B 
' i ] \ | | 
FailsM SM MM HM FullyM 
1 AM 
THU FR SAT SUN MON TUE 
aa, © © 
61° | 63° 61° 63° 66° 66° 
36 37 36 36 37° 37 
i i f i | 
| FailsM SM MM HM FullyM 
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Fully Meets Explanation 


The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users 
on mobile devices to read—it 
Fully Meets the user need. 


Note: Assume that the result 
shows current information for 
users. 


The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users 
on mobile devices to read—it 
Fully Meets the user need. 


The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users 
on mobile devices to read—it 
Fully Meets the user need. 


Remember that we consider 
weather queries, like this one, to 
be a Know Simple query for 
mobile phone users. 


Note: Assume the result shows 
current information for users. 


The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users 
on mobile devices to read—it 
Fully Meets the user need. 


Note: Assume that the result 


shows current information for 
users. 
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Query: [who is the chancellor 
of germany] 


User Location: New York, 
New York 


User Intent: Find the name 
of the current Chancellor of 
Germany. 


Query: [chevron at shoreline 
and middlefield] 


User Location: Mountain 
View, California 


User Intent: Find the specific 
gas station specified by the 
user in order to visit the 
location in person. 


Query: [decemberists crane 
wife 3 lyrics] 


User Location: Long Beach, 
California 


User Intent: See the lyrics to 
The Decemberists song 
called “Crane Wife 3.” 


Angela Merkel 


Germany, Chancellor 


Terra Bella Ave 


“nda Vista Ave 


Chevron 
Akk kk 16 reviews - 1.2 mi 


tation 


< 


Directions Share Website 


808 N Shoreline Blvd, Mountain View, CA 


94043 


More about Chevron 


The Crane Wife 3 


Song by The Decemberists 


Lyrics 

And under the boughs unbowed 
All clothed in the snowy shroud 
She had no heart so hardened 
All under the boughs unbowed 


Each feather, it fell from skin 

‘Til thread bare while she grew thin 
How were my eyes so blinded? 
Each feather, it fell from skin 


And I will hang my head 
Hang my head low 
And I will hang my head 
Hang my head low 


A gray sky, a bitter sting 

A rain cloud, a crane on wing 
All out beyond horizon 

A gray sky, a bitter sting 


And | will hang my head 
Hang my head low 
And | will hang my head 
Hang my head low 


And | will hang my head 
Hang my head low 
And | will hang my head 
Hang my head low 


And | will hang my head 
Hang my head low 

And | will hang my head 
Hang my head low, low, low 


Google Play 
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The result block displays the 
complete and correct answer 
prominently in the result block, 
in a way that is easy for users 
on mobile devices to read—it 
Fully Meets the user need. 


The intent is to visit the specific 
gas station specified by the 
user. This result block is for the 
specified Chevron location, with 
information to visit this location 
in person. 


The result has exactly what the 
user wants, displaying the lyrics 
to the specified song clearly and 
completely inside the result 
block. The user does not need 
to click to a new page to see the 
complete lyrics for this song. 


Note: The lyrics posted on 
Google Play are licensed. 
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Query and User Intent 


Query: [gas stations near 
me] 


User Location: Mountain 
View, California 


User Intent: Find gas 
stations near the user 
location. Note that this may 
be an urgent query. 


Query: [nearby coffee shops] 


User Location: Mountain 
View, California 


User Intent: Find coffee 
shops near the user location. 


BR = © Marbucks 


Red Rock Coffee 
41 kkk 41) - $$ 
Coffee Shop - Castro St 
=e p with art & live 
| 

FailsM 
Starbucks 
38 *kk** 
Coffee Shop 


nic Seattle-based coffeeho 


Clocktower Coffee Roasting 
3.9 kkk 
Coffee Shop 


jift & pastry shop with patio 


More coffee shops 
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Result Rating 
Sop, z 2 2 
Pi ¢ aF é 
2 ¥ o ra 
pulses | o, 
A Chevron 
U © 
an b ' 
B ARCO FailsM SM 
7 enes U ® 
C Shoreline Shell 
rt B U © 
More gas stations 


FullyM 


FullyM 


Fully Meets Explanation 


The intent is to find gas stations 
near the specific user location, 
represented by the blue dot on 
the map. 


This result block has a very 
satisfying list of nearby, popular, 
and prominent options within 
one mile of the specific location. 
This kind of block is especially 
helpful for users who want to 
visit the business in person. 


This result would be very 
convenient for mobile users who 
need gas immediately, 
especially for those who are 
driving and cannot easily use 
their phone. 


Note: A range is included 
because some users would be 
fully satisfied with these 
selections, while some users 
would want more information 
(e.g., gas prices, hours, distance 
from location, longer list of 
options). 


The intent is to find coffee shops 
near the specific user location, 
represented by the blue dot on 
the map. 


This result block has a very 
satisfying list of nearby, popular, 
and prominent options within a 
few miles of the specific 
location. This kind of block is 
especially helpful for users who 
want to visit the business in 
person. Note that these blocks 
are interactive—clicking on a 
coffee shop will give options to 
call the business, get directions, 
read reviews, etc. 


Note: A range is included 
because some users would be 
fully satisfied with these 
selections, while some users 
would want more information 
(e.g., hours, distance from 
location, longer list of options). 
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This block contains a map with 
the queried address, link to get 
directions, and specifies the 
landmark at the well-known 
address on the map (“The White 
House”). 


Query: [1600 pennsylvania 


: It's not unusual to search for an 
ave washington dc] 


HStNW — ate NW address (or business) that is far 
guare from the user location. Here, 
Y wv Oid'Ebbitt Grill , x the user has explicitly asked for 
Thee ae SA, |a Specific address in a far away 
_ WWashington D.C place, and this result is very 
directions, information about =e medaia : E a a 8 e 
what is located at this 1600 Pennsylvania Ave $ , Pai 
address, etc NW, Washington, DC not change the rating. 
, ` 20500 


User Location: Bakersville, 
California 


MNAS YGL 
MNJIS.WI8L 


User Intent: Find a map, 


Note: A range is included 
because some users would be 
fully satisfied with this result, 
while some users would want 
more information (e.g., photos, a 
brief description). 


; ale ? This block contains a map, link 
Aad il to get directions, departure 
times for the multiple train lines 
Query: [metro center station] 2 that operate at this station, link 
ši to a detailed schedule (i.e., the 
User Location: Adams nae “Departure board”), and an 
Morgan, Washington D.C. — option to see more information. 
Departures 
User Intent: The user wants ES) foe) EE m Since the user is located in the 
to see a map, schedule, Skay ore ER Fa SM m S Fu same city as the station, the 
transfer options, and/or other [rea : information in this block is highly 
train information for the Metro SivarSonng satisfying. 
Center station, given that the w 
user is close to the queried Wiehie Reston East eee Note: A range is included 
station. = because some users would be 
Departure board > fully satisfied with this result, 
while some users may want 
more information (e.g., nearby 
attractions, ticket prices). 


MNS MOL 


11:20AM > 


hedule information updated at 11:17 AM 
v 


More about Metro Center station 
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13.2.2 Examples of Queries that Cannot Have Fully Meets Results 


There are some queries that cannot have a Fully Meets result. Here are some examples. 


Type of Query Example = No Fully Meets Result: Explanation 


This is a broad informational query. Knitting 
is an activity anyone can do and that anyone 
can create a website for. Different users 
may want different types of content: videos, 
[knitting] instructions, patterns, etc. There is no one 
Broad queries where no official source for knitting information and no 
single result could fully one result could satisfy most users. 
satisfy all users Therefore, no Fully Meets result is possible 
for this query. 


Famous names This is a broad informational query and it is 
impossible to know exactly what the user is 
e.g., [barack obama] looking for. 


There is no dominant interpretation for this 
query. The following entities are all common 
interpretations: Americans with Disabilities 
Act, American Dental Association, and 
American Diabetes Association. While each 
interpretation has an official homepage, 
none is Fully Meets since there is no 
dominant interpretation. 


Ambiguous queries 
without a clear user intent 
or dominant interpretation 


Queries for people’s names can be tricky. 
Many or most people queries do not have a 
dominant interpretation. Even unusual 
sounding name queries may not have a 

Non-famous people names dominant interpretation. For example, the 
queries [sam wen], [tran nguyen], and [david 

e.g., [sam wen] mease] can have no Fully Meets result 
because there are multiple people with each 
of these names, and it is not clear that most 
users are looking for any one particular 
individual. 
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13.3 


Highly Meets (HM) 


A rating of Highly Meets is assigned to results that meet the needs of many or most users. Highly Meets results are 
highly satisfying and a good “fit” for the query. In addition, they often have some or all of the following characteristics: high 
quality, authoritative, entertaining, and/or recent (e.g., breaking news on a topic). 


In order to receive a Highly Meets rating, information pages such as encyclopedia articles and news articles must be 
accurate and highly credible. Highly Meets medical and scientific information pages must represent well-established 
scientific/medical consensus unless the user is clearly seeking an alternative viewpoint. For all information pages, you 
must check for accuracy and confirm that the information is supported by expert consensus where such consensus exists. 


A query can have many Highly Meets results. 


Have high standards for the Highly Meets rating. This is especially important for queries with many on-topic results. 


13.3.1 Examples of Highly Meets (HM) Result Blocks 


Query and User Intent 


Query: [trader joes] 


User Location: Charlotte, 
North Carolina 


User Intent: There are two 
possible user intents: most 
users probably want to 
visit a nearby location or 
go to the website. 


Query: [trader joes] 


User Location: Charlotte, 
North Carolina 


User Intent: There are two 
possible user intents: most 
users probably want to 
visit a nearby location or 
go to the website. 


Query: [museum of 
modern art] 


User Location: 
Manhattan, New York 


User Intent: There are two 
possible user intents: most 
users probably want to 
visit the museum (note that 
the user is located in NYC 
where the museum is 
located) or go to the 
website. 


Result Rating 
y 
Shelby “Gastonia i 
patine 
r4 
aA 
A Trader Joe's 
1133 Metropolitan Ave Ry © 
° CALL DIRECTIONS 
| ! ' | | I 
FailsM SM MM HM FullyM 
B Trader Joe's 
EA U © 
LL DIREC 
C Trader Joe's 
118 Rea R U o 
LL DIREC 
Trader Joe's 
www.traderjoes.com 
p | 'HE 
- Trader Joe's is a neighborhood FailsM SM MM HM FullyM 
grocery store with amazing food and drink from 
around the globe and around the corner 
MoMA | Museum of Modern Art 
Museum of Modern Art > 
Mobile-friendly - The Museum of Modern Art b 1 | | | l 
FailsM SM MM HM FullyM 


(MoMA) is a place that fuels creativity, ignites 
minds, and provides inspiration 
Hours & Admission - Calendar - Buy Tickets 
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Highly Meets Explanation 


This result shows a complete list 
of all three locations in the 
Charlotte area, with information 
that is especially helpful for users 
who want to visit the store. 


Note: This result block is not 
Fully Meets because users who 
want to go to the website to see 
coupons, promotions, etc. would 
have to see additional results. 


This is the official website for 
Trader Joe’s, which has highly 
authoritative information on the 
businesses’ stores, promotions, 
recipes, news, etc. 


Note: This result block is not 
Fully Meets because users who 
want information to visit the store 
have to do a bit of work to find 
that information—it is not 
immediately available. 


This is the museum’s official 
website and has very helpful 
information about exhibits, 
membership, the store, etc. 


Note: This result block is not 
Fully Meets because users who 
want information to visit the 
museum have to do a bit of work 
to find that information—it is not 
immediately available. 
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Query: [museum of 
modern art] 


User Location: 
Manhattan, New York 


User Intent: There are two 
possible user intents: most 
users probably want to 
visit the museum (note that 
the user is located in NYC 
where the museum is 
located) or go to the 
website. 


Query: [prestige bmw] 


User Location: 
Ramsey, New Jersey 


User Intent: There are two 


possible user intents: most 
users probably want to 
visit the car dealership or 
go to the website. 


Query: [prestige bmw] 


User Location: 
Ramsey, New Jersey 


User Intent: There are two 
possible user intents: most 
users probably want to 
visit the car dealership or 
go to the website. 


Museum of Modern At xr 
Museum in New York City, New York 
4.5 kkk kk 971 reviews 


U k © 


CALL DIRECTIONS WEBSITE 
9 11 W 53rd St, New York, NY 10019 
© Closing soon 10:30 am - 5:30 pm v 
Upcoming events h 


FailsM 
Evening Tours of MoMA's Mast. 


Mother's Day at MoMA 


Profiles 


Oo B 


Facebook Twitter Google+ 


v Artwork and overview 


Prestige BMW 
4.4 khk kk 138 reviews 
BMW Dealer 


© © 


DIRECTIONS WEBSITE 


{°} 985 NJ-17, Ramsey, NJ 07446 


© Closing soon 9:00 am — 8:00 pm v 


More about Prestige BMW 


Prestige BMW 
www.prestigebmw.com 


Mobile-friendly - Prestige BMW is the new, used FailsM 


& CPO BMW dealership in Ramsey, NJ near 
New York. We serve the area's BMW lease, ... 
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HM 


Fully 


The result block shows 
information about the museum, 
and is especially helpful for users 
who want to visit the museum. 


Note: This result block is not 
Fully Meets— even though there 
is a website link, it is not very 
satisfying for users who want to 
go to the website because it is 
not as informative as a web 
search result block with a title, 
snippet, and URL. 


The query is the name of a car 
dealership in New Jersey, which 
is the only dealership with that 
name in the user location 
(Ramsey, New Jersey). The 
result block shows information 
about the dealership, and is 
especially helpful for users who 
want to visit the business. 


Note: This result block is not 
Fully Meets— even though there 
is a website link, it is not very 
satisfying for users who want to 
go to the website because it is 
not as informative as a web 
search result block with a title, 
snippet, and URL. 


This is the car dealership’s 
official website and has very 
helpful information about 
inventory, financing, etc. 


Note: This result block is not 
Fully Meets because users who 
want information to visit the 
dealership have to do a bit of 
work to find that information—it is 
not immediately available. 
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Query: [belmont library] 


User Location: Belmont, 
Massachusetts 


User Intent: Find 
information about this 


library or go to the website. 


Query: [belmont library] 


User Location: Belmont, 
California 


User Intent: Find 
information about this 


library or go to the website. 


Query: [decemberists] 


User Location: Long 
Beach, California 


User Intent: Learn more 
about the band, The 
Decemberists. 


Query: [celine dion] 


User Location: Bellevue, 
Washington 


User Intent: Learn more 
about the singer Celine 
Dion. 


Channing Rd 
Belmont 


? 


Belmont Public Library 
Library in Belmont, Massachusetts 


2 reviews 


U © 


CALL DIRECTIONS WEBSITE 


Q 336 Concord Ave, Belmont, MA 02478 


© Closing soon 9:00 am — 9:00 pm v 


tion: Library 


v 


More about Belmont Public Library 


Belmont Library 
4.5 kkkh 10 reviews 
Library 


© 


DIRECTIONS WEBSITE 


1110 Alameda de las Pulgas, Belmont 
CA 94002 


© Open today 10:00 am — 9:00 pm v 


More about Belmont Library 


The Decemberists 

www.decemberists.com/ 

Mobile-friendly - New album - What A Terrible 
World, What A Beautiful World - coming January 
20, 2015. Pre-order now. 

On-Tour - Discography - “V is for Victory” Benefit 


Celine Dion | The Official Celine Dion 
Site 

www.celinedion.com/ 

Celine Dion's official website featuring Celine news, 
music, videos, album info and more! 

Videos - What's Goin' - TeamCeline account 
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j 
FailsM 


b 
FailsM 


I 
FullyM 


l 
FullyM 


There is only one library in the 
user location (Belmont, 
Massachusetts). The result 
block shows information about 
the library, and is especially 
helpful for users who want to visit 
the location. 


Note: This result block is not 
Fully Meets— even though there 
is a website link, it is not very 
satisfying for users who want to 
go to the website because it is 
not as informative as a web 
search result block with a title, 
snippet, and URL. 


There is only one library in the 
user location (Belmont, 
California). The result block 
shows information about the 
library, and is especially helpful 
for users who want to visit the 
location. 


Note: This result block is not 
Fully Meets— even though there 
is a website link, it is not very 
satisfying for users who want to 
go to the website because it is 
not as informative as a web 
search result block with a title, 
snippet, and URL. 


The user intent is too broad to 
have a Fully Meets result. 
However, this is The 
Decemberists’ official website 
and has a lot of content including 
tour dates, music videos, the 
latest album, etc. The result is 
uniquely authoritative and most 
users would be very satisfied by 
this website. 


The user intent is too broad to 
have a Fully Meets result. 
However, this is Celine Dion's 
official website and has a lot of 
content including recent news, 
show information, music, photos, 
videos, etc. The result is uniquely 
authoritative and most users 
would be very satisfied by this 
website. 
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alking Dead < 


8/10 75% 


IGN.com Rotten Tomat. 


There's a sign of the apocalypse in Los Angeles, 
where reports of a strange flu and the 
disappearance of people underscore this gritty 
drama, a prequel to AMC's uber-popular 
"Walking Dead." The story of the mysterious 
outbreak and its ramifications are told through 
the lens of high school guidance c... More 


Query: [fear the walking Network: AMC 
dead] The user intent is too broad to 
Episodes have a Fully Meets result, and 
User Location: Mountain 10. Do Not Disturb different users may be looking for 
View, California Srazot) 2 SEP 4.2018 different things. 
9. Los Muertos 1 l 
User Intent: i lac enlioeat Ful | This interactive block has a large 
Find information about the 8. Grotesque oats amount of helpful content for a 
TV show (e.g., show sae le TV show including a summary, 
summary, episode guide, > More episodes episode and cast information, 
cast information, etc.—this result would be very 
streaming options, etc.). Profiles helpful for many or most users. 


Facebook Twitter Instagram 


Alycia Debnam Cliff Curtis Frank Dillane 
Carey Travis Manawa Nick Clark 
Alicia Clark 


More about Fear the Walking Dead 


Query: [manresa reviews] The LP for this web result has 

Manresa, Los Gatos - Menu, Prices & 127 reviews for the restaurant, 
User Location: San Jose, | Reviews - TripAdvisor and is easy to use on a mobile 
California Wii tipadyiso | com/Restairant Review, f= _| phone. The first three reviews 


Akk + Rating: 4 - 127 reviews P 
FailsM FullyM 
Mobile-friendly - Manresa, Los Gatos: See 127 = uM | show automatically, and then you 


User Intent: Find reviews unbiased reviews of Manresa, rated 4 of 5 on can click to see more. This result 
for this restaurant located TripAdvisor and ranked #12 of 139... would be very helpful for many or 
in Los Gatos, California. most users. 


Query: [who is the 


chancellor of germany] is F ; 
Chancellor of Germany - Wikipedia, the This is a specific Know Simple 


User Location: New York, | free encyclopedia query for the name of the current 
N York , en.m.wikipedia.org/.../Chancellor_of_Ge... Chancellor of Germany. While 
ew Yor The current Chancellor is Angela Merkel, who is | ’ l the answer is in the description of 


4 Hi A a f FailsM FullyM 
t ffice. s Prae . 
Ueorintem Theus Pal her third term in office. She is the the web result, it is not displayed 
wants to find the name of List of Chancellors of Germany prominently ina way that is very 


the current Chancellor of easy for mobile users to read. 


Germany. 
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Query and User Intent 


Query: [broadway tickets] 


User Location: New York, 
New York 


User Intent: Research 
prices and/or purchase 
tickets to a Broadway 
show. 


Query: [kids backpacks] 


User Location: New York, 
New York 


User Intent: Research 
prices and/or purchase 
backpacks for kids. 


Query: [poison ivy] 


User Location: Atlanta, 
Georgia 


User Intent: Find pictures 
of poison ivy plants, 
information about how to 
treat poison ivy, etc. 


Query: [michael jordan] 


User Location: Boston, 
Massachusetts 


User Intent: Find 


information, news, images, 
etc. about Michael Jordan. 


Query: [dance video] 


User Location: New York, 
New York 


User Intent: Find a dance 
video to watch. 


Result 


Broadway tickets, show dates, 
calendar. Official Ticketmaster site. 
www.ticketmaster.com/broadway 

Find and buy broadway show tickets at 
Ticketmaster.com 


FailsM 


Kids’ School Backpacks & Messenger 

Bags | Free Shipping from L.L. ... 
www.IIbean.com/Ilb/.../818?...backpacks | 
Our durable kids' backpacks are loaded with FailsM 
pockets and features - like padded shoulder straps, 

nighttime reflective 


Outsmarting Poison Ivy and Other 

Poisonous Plants 

www. fda.gov/.../ucm049342.htm i 
Se 2008 - Poison ivy and other poisonous plants FailsM 
are a hazard all year round. Here are tips for 

preventing and... 


Michael Jordan - Wikipedia, the free 

encyclopedia 

en.wikipedia.org/wiki/Michael_Jordan | 
Michael Jeffrey Jordan (born February 17, 1963), FailsM 
also known by his initials, MJ, is an American 

former professional ... 


Evolution of Dance - YouTube 
m.youtube.com/watc... 
Apr 6, 2006 j 
FailsM 
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Highly Meets Explanation 


The LP for this web result offers 
many Broadway tickets for sale, 
from a highly trustworthy source. 
While you have to zoom in and 
scroll to navigate the site, this 
result would be very helpful for 
many or most users because it is 
so highly trustworthy. 


The LP for this mobile-friendly 
web result offers many kids 
backpacks for sale. The 
company is well-known for 
manufacturing and selling high 
quality backpacks for kids of 
different ages. There is a lot of 
product information and many 
user reviews for each backpack, 
in addition to filter options that 
are easy to use on a phone. This 
result would be very helpful for 
many or most users. 


The LP for this mobile-friendly 
web result is part of an 
authoritative government website 
and provides a lot of helpful 
information about poison ivy. 
This result would be very helpful 
for many or most users. A few or 
some users may wish to see 
additional results. 


The LP of this web result is a 
mobile-friendly Wikipedia article 
about Michael Jordan. This 
result would be helpful for many 
or most users. 


Users are looking for good or 
entertaining dance videos and 
there are many options online. 


There are many on-topic results 
for this query and this is one very 
popular video of a comedian 
demonstrating dance styles from 
previous decades. 
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Query: [purple 
coneflowers] 


User Location: New York, 
New York 


User Intent: Find images 
of or information about 
purple coneflowers. 


Query: [invasive species] 


User Location: St. Louis, 
Missouri 


User Intent: Find more 
information about invasive 
species. 


Query: [turmeric] 


User Location: 
Sunnyvale, California 


User Intent: Find 
information about the 
spice or the restaurant in 
Sunnyvale. 


Query: [how to get from 
seattle to xpppzyfii] 


User Location: Seattle, 
Washington 


User Intent: Get directions 
from Seattle to another 
place. Note that the 
location in the query 
(“xpppzyfii”) is garbled and 
itis impossible to know 
what the user wanted. 
(The name was likely 
mistyped or mistranscribed 
from a voice query.) 


Images 


More images for purple coneflowers 


An invasive species is a 

plant or animal that is not 

native to a specific 

location (an Introduced www nature.org 
species); and has a tendency to 
spread, which is believed to cause 
damage to the environment, human 
economy and/or human health. 


FailsM 


Invasive species - Wikipedia, the free 
encyclopedia 
en.m.wikipedia.org/wiki/Invasive_species 


Sunnyvale Caltrain} 


Olson Way, q 


Dishdash ro 


a 


Turmeric Restaurant 
3.7 kkk k k 207 reviews 


$$ +» Event Venue 
h Murphy Avenue, Sunnyvale, CA 94086 
00 
> Open today - 11:30 am — 2:00 pm, 12:00 — 2:30 pm 
5:30 — 10:00 pm, 5:30 — 10:00 pm 


(4 


N Y © 


Call Directions Website 
Menu: turmericrestaurant.com 


More about Turmeric Restaurant 


No results for that place. Try entering it 
below to get suggestions. 


Seattle, WA 
FailsM 
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This block contains many helpful 
images, and the landing page 
shows even more images—it 
Highly Meets the user intent. 


The result block has a good 
summary of what invasive 
species are, but some users 
would probably want to see more 
information like examples, 
impact, etc. 


For users in Sunnyvale, 
California, the query [turmeric] 
could have two different 
interpretations: a popular 
restaurant named Turmeric or 
the spice turmeric. Because the 
user is located in Sunnyvale, 
many mobile users are probably 
looking for the restaurant. 
However, it is not completely 
clear because some users may 
want to find information about the 
spice. This result about the 
restaurant has satisfying 
information for users who want to 
visit —it Highly Meets the most 
likely user need. 


Even though the user has to 
re-enter the destination, the 
result block recognizes that the 
user was asking for directions 
from Seattle. The user has to 
enter the destination manually 
because the name in the query 
(“xpppzyfii’) is so garbled that 
there is no good guess as to 
what the user meant. 


Therefore, this block is satisfying 
because it helps the user correct 
the request and then immediately 
get the desired directions. 
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Query: [kristen wiig] 


User Location: New York, 
New York 


User Intent: Find more 
information about the 
actress, comedian, writer, 
and producer. 


Query: [greek alphabet] 


User Location: Albany, 
New York 


User Intent: Find historical 
information about the 
Greek alphabet, the 
names of the letters in the 
Greek alphabet, images of 
the letters, etc. 


Query: [jennifer aniston] 


User Location: Dover, 
Delaware 


User Intent: Find images, 
recent news, gossip, 
information, etc. about 
Jennifer Aniston. 


Query: [seattle, 
washington] 


User Location: Arlington, 
Texas 


User Intent: Find 
information, news, maps, 
etc. related to Seattle, 
Washington. 


Kristen Wiig | www.kristenwiig.org -° 
Your fan site for Oscar ... 
www.kristenwiig.org 

Mobile-friendly - Watch the full interview on the 
official website to hear Kristen and director 
Shira Piven talk about their careers in acting 
Photo Gallery - KristenWiig.org Video Archive 


Images 


ap ydé 
ThB 
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More images for Greek alphabet 


In the news 


What Oscar Snubs? Inside Jennifer 
Aniston and Angelina Jolie's Night at 
the Critics’ Choice Awards : 
People.com 


People.com - 5 hours ago 


EXCLUSIVE: Jennifer Aniston double 
heartbreak as BOTH her ‘lost love’ and best 
friend, who starred in cult classic ... 


Daily Mail - 2 hours ago 


Jennifer Aniston Reacts to Oscars 2015 
Cake Snub, Says Reese Witherspoon and 
Julianne Moore Emailed Her 


Us Magazine - 4 hours ago 


More news for jennifer aniston 


agrnamnish 
qua 


lon! o's 


Seattle 
City in Washington 


Weather: 52°F (11°C), Wind 0 mph (0 km/h), 88% Humidity 


Points of interest 


Pike Place Space Needle Experience 
Market Music Project 
Museum 


Upcoming events and overview 
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This is a fan site dedicated to 
Kristen Wiig. The website has 
comprehensive info including 
over 50,000 pictures and 300 
video clips, interviews, articles, 
and more. This website Highly 
Meets the need of many or most 
users. 


This is a broad query. Different 
users may be looking for different 
things. 


This result shows multiple tables 
of images with the letters in the 
Greek alphabet. This result 
would be helpful for many users. 
Some users may wish to see 
additional results. 


This is a broad query for an 
actress. Different users may be 
looking for different things. 


This result shows recent news 
about Jennifer Aniston. The 
articles are timely (at the time 
this was written) and relatively 
interesting, and not just common 
everyday news about the 
actress. Many users would be 
satisfied with the result and some 
users may wish to see more 
results. 


This is a broad query for a city. 
Different users may be looking 
for different things. 


The result contains a wide 
variety of helpful information 
such as a map, weather 
information, several points of 
interest, and has an option to see 
more information containing 
nearby events and facts about 
the city. Moderately Meets+ to 
Highly Meets is an appropriate 
rating for this result. 
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Query: [emily blunt] 


User Location: Portland, 
Oregon 


User Intent: Find 
biographical information 
(e.g., filmography) or 
current news/celebrity 
gossip about Emily Blunt. 


Query: [coldplay yellow 
lyrics] 


User Location: Billings, 
Montana 


User Intent: Find the 
lyrics to the song 
“Yellow”, which was 
written by the band 
Coldplay. 


Query: [1969 moon 
landing] 


User Location: New 
Haven, Connecticut 


User Intent: Find accurate 
and authoritative 
information about this 
historical event. 


Emily Blunt 
Actress 


OVERVIEW MOVIES 


HF 


Emily Olivia Leah Blunt is an English actress. She 
made her professional debut in The Royal Family. 
Her screen debut was in the 2003 television film 
Boudica. Wikipedia 


Spouse: John Krasinski (m. 2010) 


Movies 


Edge of Sicario The Huntsm: 
Tomorrow 2015 Winter's War 
2014 2016 


More about Emily Blunt 


Coldplay: Lyrics: Yellow 
Coldplay > recordings > parachutes > yell... 


The official Coldplay website, featuring news, lyrics, tour dates, 
blogs, interviews, pictures and videos. ... Yellow Look at the stars 
Look how they shine for you And everything you do Yeah, they 
were all yellow | came... 


July 20, 1969: One Giant Leap For Mankind | NASA 
NASA (.gov) > apollo > apollo11 


At 10:56 p.m. EDT Neil Armstrong is 
ready to plant the first human foot on 
another world. With more than half a 
billion people watching on television, he 
climbs down the ladder and proclaims: 
"That's one small step for a... 
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This is a broad query for an 
actress. Different users may be 
looking for different things. 


The result contains a variety of 
helpful information such as 
images, brief biography, recent 
movies, and an option to see 
more. Moderately Meets+ to 
Highly Meets is an appropriate 
rating for this result. 


This page provides the 
requested lyrics on Coldplay's 
official website. Users can 
also navigate around the site 
to listen to the song, explore 
all albums the song appears 
on, listen to or view lyrics for 
other songs on the same 
album, or learn more about 
the artist overall. 


Additionally, users can trust 
that the lyrics on the LP are 
100% accurate due to the 
uniquely authoritative nature 
of the source, which makes 
this result very helpful for 
many or most users. 


The LP for this web result 
contains authoritative information 
about the Apollo 11 moon landing 
on the official website of NASA, 
which was the organization that 
oversaw this mission in 1969. 
The MC has a lot of helpful 
information and content, 
including a detailed summary, 
video footage, and images of the 
moon landing that users can 
view and explore. This result 
would be very helpful for many or 
most users. 


101 


Query and User Intent 


Query: [stem education] 


User Location: Memphis, 
Tennessee 


User Intent: Find 
information or recent news 
about education in STEM 
fields (i.e., Science, 
Technology, Engineering, 
and Mathematics). 


Query: [mentos and coke 
death] 


User Location: Las 
Vegas, Nevada 


User Intent: Find accurate 
information about a rumor 
that a mixture of Mentos 
and Coca-Cola can cause 
death, which was spread 
on the Internet in 2006. 


Query: [pj fleck] 


User Location: 
Minneapolis, Minnesota 


User Intent: Find 
information or recent news 
about P.J. Fleck, who had 
just become the new head 
football coach at the 
University of Minnesota a 
few days before this 
example was written. 


Query: [batman lego 
movie review] 


User Location: Orlando, 
Florida 


User Intent: Find reviews 
of the 2017 film The LEGO 
Batman Movie on credible 
news or entertainment 
sources. 


Result 


Educators, advocates see increased importance 
in US STEM... 
Fox News > 2015/06/29 > educators-adv... 


AMP - Jun 29, 2015 - When it comes to education 
reform in the United States, Science, Technology, 
Engineering, and... 


Mentos and Coke Death : snopes.com 
Snopes > Fact Check» Horrors 


Nov 9, 2006 - Mixture of Mentos and Coca-Cola kills child? 
.. Last week a little boy died in Brazil after eating MENTOS 
and drinking COCA COLA together. 


Welcome aboard, ‘life coach’ PJ. Fleck - 
StarTribune.com 
Star Tribune > welcome-aboard-life-coac... 


Jan 9, 2017 - P.J. Fleck's tenure as head football coach 
at the University of Minnesota will not be defined by an... 


‘The Lego Batman Movie' is the best Batman 
movie since 'The Dark Knight’ - LA Times - Los 
Angeles Times 

Los Angeles Times > movies > la-et-mn4... 


AMP - Feb 9, 2017 - Justin Chang reviews The Lego 
Batman Movie; starring Will Arnett. 


Rating 
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Highly Meets Explanation 


This is an informational query on 
a fairly broad topic. Different 
users may be looking for different 
things, yet accuracy and 
credibility of the source are 
important. 


This LP is a news article 
published on a reputable news 
website. The MC of the article 
covers recent efforts to reform 
STEM education in the U.S., and 
it includes citations from expert 
sources. This result would be 
helpful for many or most users. 


This LP is on a well-known 
fact-checking and debunking 
website. The page calls this 
story "False", then provides a 
credible explanation of how the 
authors came to that conclusion, 
including history of similar urban 
legends. This result would be 
very helpful for users seeking to 
verify the credibility of this rumor. 


This LP is an opinion piece 
written by the Editorial Board of 
the largest newspaper in the U.S. 
state of Minnesota. The MC of 
the article expresses 
well-informed opinions about 
Fleck's upcoming tenure as head 
coach, and it would likely be of 
interest to many or most users 
when this query was issued. 


This LP is an in-depth review of 
the queried movie by a film critic 
at a reputable newspaper. This 
result would be very helpful for 
many or most users. 
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13.4 Moderately Meets (MM) 


A rating of Moderately Meets is assigned to results that are helpful and satisfying for many users or very satisfying for 


some users. 


Moderately Meets results have fewer valuable attributes than Highly Meets results. Moderately Meets results should 
still “fit” the query, but they might be less comprehensive, less up-to-date, come from a less authoritative source, etc. 


Moderately Meets results are not low quality, out-of-date, or inaccurate. Moderately Meets results are generally average 


to good. 


13.4.1 Examples of Moderately Meets (MM) Result Blocks 


Query and User Intent 


Query: [shutterfly] 


User Location: Boise, 
Idaho 


User Intent: Go to the 
Shutterfly website or find 
information about the 
company. 


Query: [tom cruise] 


User Location: 
Scottsdale, Arizona 


User Intent: Find 
biographical information or 
current news/celebrity 
gossip about Tom Cruise. 


Query: [seattle, 
washington] 


User Location: Denver, 
Colorado 


User Intent: Find 
information, news, maps, 
etc. related to Seattle, 
Washington. 


Query: [city of angels] 


User Location: 
Los Angeles, California 


User Intent: Find 
information about the 
movie, song, musical, 
school, or some other 
entity by this name. 


Result 


Shutterfly | CrunchBase 
www.crunchbase.com/.../shutterfly 

Shutterfly is a manufacturer and digital retailer, 
offering personalized stationery and other related 
products and services 


j 
FailsM 


Tom Cruise Pictures, Biography, 
Filmography, News ... - Starpulse.com 
www.starpulse.com/Actors/Cruise, Tom/ 

Tom Cruise Pictures, Biography, Filmography, 
News, Box Office, Videos, Awards 


FailsM 
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Seattle, WA 


Ñiàp daté ©2014 Google 


City of Angels Independent 
Studies School } 


FailsM 
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Call Directions Website 
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Rating 


Moderately Meets 
Explanation 


The LP for this web result is a 
high-quality page on 
crunchbase.com—a member site 
on the TechCrunch 
network—with a profile of 
Shutterfly. The MC has a lot of 
helpful and accurate information 
about the company and is easy 
to read on a mobile device. This 
result would be helpful for many 
users or very helpful for some. 


The LP for this web result 
contains information about Tom 
Cruise, which would be helpful 
for many users or very helpful for 
some users. The photos and 
main links can easily be 
accessed on a phone. 


Note that there are many, many 
pages about Tom Cruise and this 
result is not helpful enough for a 
rating of Highly Meets. 


This is a broad query for a city. 
Different users may be looking 
for different things. 


The result shows a map of 
Seattle, Washington and clicking 
on the map brings up a more 
detailed map. This result may be 
helpful for some users. 


This result reflects one of several 
possible user intents, as the 
query is also the name of a 
movie, song, musical, and more. 
However, since the user location 
is near the school, this result 
would be very helpful for some 
users—specifically, users who 
are looking for the school. 
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User Intent: Find more 
information about the 
actress, comedian, writer, 
and producer. 


Query and User Intent | Result Rating 
Query: [nicaraguan 
money] 
User Location: Miami, . , 
Florida Nicaraguan córdoba | „o “EPERE 
R . : FailsM SM MM HM FullyM 
qua, 
User Intent: Find out what 
Nicaraguan currency is 
called, what it looks like, 
the exchange rate, etc. 
Query: [where is virginia in | Virginia is bordered by Maryland 
the us] and Washington, D.C. to the north 
and east; by the Atlantic Ocean to 
g PEN the east; by North Carolina and 
User Location: Louisville, Tennessee to the south; by k i a} 
Kentucky Kentucky to the west; and by West eee roe an HM Fum 
Virginia to the north and west. 
User Intent: Find where Virginia - Wikipedia, the free 
the state of Virginia is ile ee 
i en.m.wikipedia.org/wiki/Virginia 
located in the U.S. e 
See results about 
Query: [mercury] 
Mercury (Chemical Element) e] 
5 . Symbol: Hg 
User Location: New Electron configuration: [Xe] 4714 5.. 
Haven, Connecticut 
Mercury (Planet) EAS } f =| f I 
User Intent: Find Radius: 1,516 miles (2,440 km) r FailsM SM MM HM FullyM 
A . Surface area: 28.88 million sq mile 
information on the 
chemical element, the Mercury (Automobile company) G 
planet, or the automobile Founder: Edsel Ford E 
company. Founded: 1939 
Query: [every breath you 
take lyrics] 
y STING LYRICS - Every Breath You Take 
User Location: Fort - A-Z Lyrics 
Davis, Texas www.azlyrics.com/lyrics/sting/everybreat... h 1 t f l 
Lyrics to "Every Breath You Take" song by STING: FailsM SM MM HM FullyM 
y ‘ Every breath you take Every move you make Every 
User Intent: Find the lyrics | bond you break... 
to the song “Every Breath 
You Take,” which was 
written by Sting. 
Query: [kristen wiig] 
Kristen Wiig! The Official Website of 
User Location: New York, | “"ste" Wiig. 
kristenwiig com 
New York lobile-friendly - The official website of Kristen t =|) i s | 
Wiig. Kristen Wiig is not on Twitter, Facebook. FaileM SM MM HM FullyM 


MySpace, or any other social networking 
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Moderately Meets 


Explanation 


This result displays the name of 
Nicaraguan currency. Some 
users may have their need met 
by learning the name of the 
currency, but many users may 
want images, know the exchange 
rate, etc. This result would be 
helpful for many users or very 
helpful for some users. 


The description in this result 
block may be helpful for users 
who are already familiar with this 
area in the United States. Many 
or most users would probably 
want to see a map or a clearer 
description. 


This block contains a brief 
description of different 
interpretations for mercury. 
Clicking on the links will take the 
user to the search results page 
for that interpretation. The links 
may save users from doing 
follow-up searches to clarify the 
correct interpretation, but this 
can be difficult on a phone. 


While the descriptions are brief, 
the links are a helpful feature, so 
the result Moderately Meets the 
user intent. 


The LP for this lyrics website 
page has the requested lyrics. 
There are many lyrics websites 
on the web and many pages are 
not 100% accurate. Moderately 
Meets is an appropriate rating for 
average pages with the 
requested lyrics. 


This is Kristen Wiig’s official 
website, but the only content is 
links to her talent and PR 
agencies, along with a note that 
says she is not on any social 
networking sites. While this 
information is helpful, many 
users would want to see more. 
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13.5 Slightly Meets (SM) 


A rating of Slightly Meets is assigned to results that are helpful and satisfying for some or few users. Slightly Meets 
results may serve a minor interpretation, be low quality, have stale or outdated information, have some minor 
inaccuracies, be too specific, too broad, too niche for the general user population considering the query, etc. to receive a 


higher rating. 


Important note: On mobile phones, the "cost of a click" is high. Many users decide which result to click or tap on based 
on the title of the web result on the search results page. (Note that these titles usually come from webpages.) A result 
with a very misleading or exaggerated title should be rated Slightly Meets or lower, due to the poor user experience that 
occurs when the landing page does not match the expectation of the user when clicking or tapping on the result. 


13.5.1 Examples of Slightly Meets (SM) Result Blocks 


Query and User Intent 
Query: [britney spears] 


User Location: Baltimore, 
Maryland 


User Intent: Find 
information about Britney 
Spears (e.g., current 
news, pictures). 


Query: [honda odyssey] 


User Location: Miami, 
Florida 


User Intent: We will 
assume that users are 
interested in the current 
Honda Odyssey model, 


unless specified otherwise. 


Query: [hot dog] 


User Location: 
Sunnyvale, California 


User Intent: Find 
information about hot 
dogs, such as recipes or 
nutrition information. 


Query: [motorcycles] 


User Location: Potsdam, 
New York 


User Intent: Find 
information about 
motorcycles. 


Result 


Britney Spears Files for Divorce | 
TMZ.com 
www.tmz.com/.../britney-spears-files-for- 

Britney Spears has filed a petition for 
divorce from Kevin Federline. TMZ obtained the legal 


j 
FailsM 


papers, ... 


2010 Honda Odyssey - Kelley Blue 
Book 
www.kbb.com/honda/odyssey/2010-hon... 

Bx Ed 01 
Learn more about the 2010 Honda Odyssey with 
Kelley Blue Book expert reviews. Discover 
information including 


Hot Dog... The Movie (1984) - IMDb 
www.imdb.com/title/tt0087425/ 

Directed by Peter Markle. With David Naughton, 
Patrick Houser, Tracy Smith, John Patrick Reger. 


j 
FailsM 


Naive corn-pone ... 


} 


n 


state Farm: Drew 
Ty McFadden IV 


Ai 
72 Sovie's Cycle Shop 

Sovie's Cycle Shop I 

lo revie 


i t FailsM 


State Farm: Drew McFadden IV t 
No reviews 


e Agen arket S CALL 
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Rating 


i ' | 
SM MM HM FullyM 
j } | 
SM MM HM FullyM 
= ' | 
SM MM HM FullyM 
| | 
SM MM HM FullyM 


Slightly Meets Explanation 


The LP of this mobile-friendly 
web result has a 2006 article 
about Britney Spears filing for 
divorce. This is very old, stale 
news, making the result helpful 
for few users. 


Although the LP has 
comprehensive information from 
a reputable source, and is easy 
to use on a mobile device, the 
information is about the 2010 
Honda Odyssey. This 
information would be considered 
stale for the query today, making 
this result helpful for some or few 
users. 


The LP of this web result is about 
the movie “Hot Dog,” which came 
out in 1984 and is a minor 
interpretation of this query. 
Because this is an unlikely 
interpretation, this web result 
would be helpful for few users, 
even though the landing page 
has good content from a 
reputable source, and is easy to 
use on a mobile device. 


This result provides contact 
information for one of the two 
motorcycle shops in Potsdam, 
New York. It is possible this 
would be helpful for some users, 
but many users would be looking 
for information. Additionally, 
there is distracting 

information (a listing for an 
insurance agent). 
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Q : Tabe lincoln’ This is a Wikipedia page that has 
luery: [abe lincoln's List of Presidents of the United birthdays for all U.S. presidents, 
birthday] States by date of birth - Wikipedia nduding Abraham Lineal: 


However, his birthday is not 
prominently displayed and it is 
U.S. Presidents, organized by date of i obvious ish D a ed IS 

, , i itional li i ere—users have to do some 
User Intent: Find this birth, plus additional lists of birth related 
specife piece of statistics work and search around on the 
ss a United States Presidents by page to find the answer, making 
: SM to SM+ an appropriate rating. 


User Location: Fort en.m.wikipedia.org/.../List_of_Presidents... 
Davis, Texas Mobile-friendly - The following is a list of 


Query: [lack of sex and The quality of writing in this 

problems with my article, which was created by a 

marriage] 5 Tips to Fix a Sexless Marriage Or person without expertise in 
Relationship - Ezine Articles I marriage or relationship 

ao Ee ezinearticles.com > ... > Marriage =a . . 

User Location: Waco, Mobile-friendly - Feb 25, 2008 - Look tothe past- | FailsM counseling, is poor. Even though 

Texas Sexless marriages don't just spring up out of thin the article is about the query, the 
air. They are just symptoms of other ... page is low quality and 

User Intent: Find help for untrustworthy. It would be helpful 


marital issues. for few users. 


Stevens Creek Blvd 


Santana Row |) 
Judro Way 


- Tnai Burke Williams 
Query: [nail spa] uo ee Burke Williams is a day spa that 


User Location? San Josè Day Spa offers many treatments like 

California ` : = massage therapy, skin care, etc. 
° © FailsM While it does have nail care 

DIRECTIONS WENNE options, it is not a dedicated nail 


User Intent: Find a nail 
spa. 


spa in or near San Jose. 9 355 Santana Row #2010, San Jose, CA 
95128 


© Open today 8:00 am — 10:00 pm v 


More about Burke Williams 


Query: [what type of Although there are a 
sharks live in rivers] handful of species that 
live in freshwater, we tend This result block does not 
. : to think of sharks as being A . . . 
User Location: Tucson, ocean dwellers: Put there Babna contain enough information to be 
Arizona very unique shark that can live in a | fully satisfying and mentions only 
both ocean and fresh water Fan one type of shark. Most users 
User Intent: Find estvaniesand nie Bull Shark. would need to do further 
. . Shark Savers :: How Bull Sharks 
information about what Survive in Firean Wator research. 
types of sharks live in www.sharksavers.org/.../how-bull-sharks- 


rivers. 


The answer in this result block is 
Most of Bexar County = . 
stayed in 210, while the = . -` really confusing. Even if users 
San Antonio suburbs and =>- > can figure it out, they would likely 


the Texas Hill Country "NA i í i 
User Location: Austin, were split off as area code 830. The still have A click on the link, a 
Texas southern portion became area code io go to anot er website, to confirm 
956. The 1997 split made San aa the information. 
. Antonio one of the few cities that 
User Intent: Find what are split between two area codes. While-thereis:aiman: voui would 
area uses the 210 area Area code 210 - Wikipedia, the free fave toexoand or pins on itto 
encyclopedia 
code for phone numbers. en.m_wikipedia_org/wiki/Area_code_210 see the details so it is not that 


helpful. 


Query: [210 area code] 
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Query and User Intent 


Query: [ibm] 


User Location: Rockville, 
Maryland 


User Intent: Go to the IBM 
website or find information 
about the company. 


Query: [where is nibiru] 


User Location: Oklahoma 
City, Oklahoma 


User Intent: Find the 
location of "Nibiru", which 
was a fictional planet in 
the movie Star Trek Into 
Darkness, as well as a 
pseudoscientific outer 
planet in the solar system 
that doesn't actually exist. 


Query: [ellen degeneres 
news] 


User Location: Anaheim, 
California 


User Intent: Find recent 
news about the 
comedian/actress Ellen 
DeGeneres. 


FS ee 
FailsM FullyM 


> More images for ibm 


Planet Nibiru has appeared in the 
sky and DOOMSDAY is on the way 


https://www.thesun.co.uk/.../planet-ni 
biru-has-appeared... 

Aug 23, 2016 - Now a YouTube channel 
called Nibiru/Planet X has uploaded 
footage shot in August which allegedly 
shows a blood red moon alongside a ... 


I 
FullyM 


Shocking! Ellen's Goodbye... 
www.dailymail.co.uk/.../article-2357877/... h 1 ——+ 
7 j 2013 = Leaked secret has fans 


FailsM HM FullyM 


outraged! Ellen has lied for years ... 
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Slightly Meets Explanation 


This block contains images of the 
logo for IBM, but these images 
are not particularly helpful for this 
query. 


The title of this article makes it 
appear as though the article 
describes an impending 
catastrophe. Yet, the MC states 
in its third paragraph that Nibiru 
"doesn’t actually exist" and calls 
believers in it "conspiracy 
theorists". The title contradicts 
the content of the article, and as 
a result could mislead users who 
don't read the full article. 


Note: The "cost of a click" is high 
on mobile phones. Results with 
very misleading or exaggerated 
titles should be rated Slightly 
Meets or lower. 


While the title of this article 
implies that something bad or 
outrageous happened involving 
Ellen, the actual content of the 
article is about an anti-aging 
cosmetic that she is claimed to 
have used. The content of the 
page is not related to what the 
misleading and exaggerated title 
suggests, so it should be rated 
Slightly Meets or lower. 
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13.6 Fails to Meet (FailsM) 


A rating of Fails to Meet should be assigned to results that are helpful and satisfying for no or very few users. Fails to 
Meet results are unrelated to the query, factually incorrect (please check for factual accuracy of answers), and/or all or 
almost all users would want to see additional results. These results completely fail to meet the user intent, such as a lack 
of attention to an aspect of the query (or user location) that is important for satisfying user intent. Fails to Meet may also 
be used for results that are extremely low quality, have very stale or outdated information, be nearly impossible to use on 


a mobile device, etc. 


Fails to Meet should be used for the following types of Lowest Page Quality results: 

e Results that harm users (e.g., malicious download pages, "phishing" pages or websites, etc.). 

e Pages or websites that misrepresent the website's owner or purpose, such as by impersonating a different site 
(e.g., copied logo or branding of an unaffiliated site, URL that mimics another site's name). 

e Pages or websites that appear to be deliberate attempts to misinform or deceive users by presenting factually 
inaccurate content (e.g., fake product reviews, demonstrably inaccurate news, etc.). 

e Pages or websites with factually inaccurate content that can cause harm to users, regardless of their purpose or 
intent (e.g., inaccurate medical information; inaccurate information about how to vote, obtain a visa or green card, 
pay government taxes, etc.). 


The following should also be rated Fails to Meet because they lead to very poor and upsetting user experiences: 
e Porn results for non-porn-seeking queries. 
e Upsetting or offensive results for queries that are not obviously seeking upsetting or offensive content. 
e Pages that directly contradict well-established scientific or medical consensus for queries seeking scientific or 
medical information, unless the query indicates the user is seeking an alternative viewpoint. 
e Pages that directly contradict well-established historical facts (e.g., unsubstantiated conspiracy theories), unless 
the query clearly indicates the user is seeking an alternative viewpoint. 


13.6.1 Examples of Fails to Meet (FailsM) Result Blocks 


Query and User Intent 
Query: [dogs] 


User Location: Pittsfield 
Charter Township, 
Michigan 


User Intent: Find 
information (e.g., pictures, 
breeds, training details, 
etc.) related to dogs. 


Note: For this query, it is 
very unlikely users want to 
go anywhere in person. 


Query: [mike] 


User Location: Eustis, 
Florida 


User Intent: Find 
information about 
something related to Mike. 


Note: For this query, it is 
unlikely users want to go 
anywhere in person. 


Result Rating 
Arbor Dog Day Care 
4.5 kkkk* (7) Re 
Pet Sitter - S Main St ae 
Open un 
Camp Bow Wow = ji = $ t l 
45 khkk (38) 2 FailsM SM MM HM FullyM 
Dog Da re Center te Cil = 
Sunset Animal Care LLC 
N U 
F CALL 
OF m 
More dog 
Mike Bender Golf Academy 
Nor E ' | ' I 
Private Golf Course » Shadowmoss Cir CALL FailsM SM MM HM FullyM 
Closed now 
Mike Holt Enterprises, Inc 
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Fails to Meet Explanation 


This is a broad informational 
query for an animal. Different 
users may be looking for different 
things. 


This result block shows 
visit-in-person information for 
three different dog care providers 
near the user location. However, 
the query is very broad and it is 
unlikely users want to go 
anywhere in person. 


This is a broad query and it is 
unclear exactly what the user is 
looking for. 


This result block shows 
visit-in-person information for two 
businesses that contain the 
name Mike. However, the query 
is very broad and it is very 
unlikely these businesses are 
what users seek given the query. 
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Query: [tampa rays] 


User Location: Tampa, 
Florida 


User Intent: Find 
information (e.g., scores, 
roster, recent news) about 
a professional baseball 
team in Florida called 
Tampa Bay Rays. 


Note: For this query, it is 
unlikely users want to go 
anywhere in person. 


Query: [wral] 


User Location: Wake 
County, North Carolina 


User Intent: Go to the 
WRAL website or find 
other information about the 
news station in North 
Carolina. 


Note: Users generally 
would not go to a news 
station in person. 


Query: [texas farm 
fertilizer] 


User Location: Dallas, 
Texas 


User Intent: Find 
information about, or 
where to buy, fertilizer from 
the company Texas Farm. 


© 


DIRECTIONS WEBSITE 


200 Madison St, Tampa FL 33602 


© closed now Hours v 


More about Bob’s Cards 


4.2 KKK 


Texas Farm Products Co 


Farm 


U © © 


CALL DIRECTIONS WEBSITE 


9 915 S Fredonia St, Nacogdoches, TX 
75964 


@ Claim this business 


More about Texas Farm Products Co 
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FailsM 


This result block shows 
information to visit a sports card 
store in person, which happens 
to have the baseball team in the 
name. However, remember that 
for this query, it is unlikely users 
want to go anywhere in person. 


Note: This business was 
fabricated for the purposes of 
this example, and does not exist 
in real life. 


WRAL is a television news 
station in North Carolina. Users 
want to go to the website or find 
information about the station. 


In this example, clicking on the 
blocks show information about 
the businesses’ corporate 
headquarters. Note that 
television stations (like radio 
stations, newspapers, etc.) 
usually have special contact 
information for the public that is 
different than their corporate 
headquarter information. Also, 
with the query just being the 
name of the station, it is very 
unlikely that users want this kind 
of information. 


This result block shows the 
Texas Farm corporate office, 
which is not helpful given the 
user intent—users could not 
physically go to the company’s 
corporate office to buy or learn 
about the product. Fertilizer is 
the kind of product that you 
would have to buy at a store. 
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Query: [united 656] 


User Location: Skokie, 
Illinois 


User Intent: Find 
information about flight 
656 operated by United 
Airlines (e.g., whether the 
flight is on time, what gate 
it is departing from). 


Query: [german cars] 


User Location: Toledo, 
Ohio 


User Intent: Find 
information about German 
cars or go to the official 
homepage of a German 
automaker. 


Query: [company to get rid 
of the possum in my attic] 


User Location: Naperville, 
Illinois 


User Intent: Finda 
company to trap and 
remove a possum from the 
attic. 


Query: [starting jets 
quarterback 2001] 


User Location: Chicago, 
Illinois 


User Intent: Find the 
name of the starting 
quarterback for the New 
York Jets football team in 
2001. 


Query: [navigate to 
yellowstone national park] 


User Location: Denver, 
Colorado 


User Intent: Navigate to 
Yellowstone National Park 
from the user’s location. 


Se ales Unitarian © rch), 1 


Second Unitarian Church 
1 review 
Church - W Barry Ave 
FailsM 
O'Hare International Airport 
3.6 kk IH * (1,007) 
Airport - W O'Hare 
Massive airport serving the Midwest 


Catalyst Ranch 
46 kk aH (11 


Event Venue - W Randolph St 


Subaru of America: Subaru Cars, 
Sedans, SUVs, Crossovers, Wagons 
https://www.subaru.com/ 

Visit Subaru of America for reviews, pricing and 
photos of Subaru Cars, Sedans, Wagons, 
Crossovers & SUVs. 


Complete Pest Control - Adelaide 
South Australia | Adelaide Pest ... 
completepest.com.au/ 

Our family owned business has been delivering 
professional pest control services to metropolitan 
Adelaide since 1988 

Termite Treatments - Pest Specials 


New York Jets 2013 Team Player 
Roster - ESPN 
espn.go.com/nfl/.../roster/_/.../new-york-... 
Complete New York Jets team roster on ESPN.com, 
including all players, positions, and jersey numbers. 


FailsM 


Yellowstone National Park (U.S. 
National Park Service) 

www.nps.gov/yell/ 

National Park Service official site with useful 
information for the park visitor and background on 
the natural and scientific ... 

4.6 kkk kk 602 reviews 

Plan Your Visit - Webcams - Park Roads 


FailsM 
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FullyM 


This result shows information for 
a church, an event venue, and 
the closest airport. However, 
users clearly want to know 
details about a specific flight, and 
there is absolutely no information 
in the block about the flight. This 
result Fails to Meet the user 
intent. 


This LP is the homepage of 
Subaru, a Japanese car 
company, not a German car 
company. 


The page completely Fails to 
Meet the user intent and would 
be helpful for no or very few 
users. 


This LP is the homepage of a 
pest control company in 
Australia. U.S. users would need 
a U.S. company to take care of 
this problem. There is a 
mismatch between the page and 
the locale that makes this result 
helpful for no users—it 
completely Fails to Meet the 
user intent. 


Although this is a trustworthy 
website for information about 
NFL football, this LP does not 
contain the information 
requested by the user. This 
result completely Fails to Meet 
the user intent. 


This web result (the official 
homepage of Yellowstone 
National Park) completely fails to 
respond to the specific Device 
Action query: to navigate to the 
park. 
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